00000100:00080000:2.0:1787756177.494633:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756177.494637:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654671744:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756177.494642:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654671872:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756177.494659:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756177.494667:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756177/real 1787756177] req@ffff8c05a9f78700 x1874588654671872/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756177.494675:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588654671872/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756177.494680:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756177.494682:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756177.494684:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654671872:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756177.494694:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654672000:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756177.494704:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756177.494710:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756177/real 1787756177] req@ffff8c05a9f78000 x1874588654672000/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756177.494724:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78000 x1874588654672000/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756177.494729:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756177.494731:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756177.494733:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654672000:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756177.494736:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654672128:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756177.494748:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756177.494754:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756177/real 1787756177] req@ffff8c05a9f79c00 x1874588654672128/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756177.494761:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79c00 x1874588654672128/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756177.494768:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756177.494771:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756177.494774:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654672128:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756177.494780:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654672256:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756177.494811:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756177.494831:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756177/real 1787756177] req@ffff8c05a9f7b480 x1874588654672256/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756177.494843:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588654672256/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756177.494848:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756177.494850:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756177.494852:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654672256:192.168.201.107@tcp:8:kworker.0 00010000:00010000:2.0:1787756181.142036:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756182.166066:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787756182.614109:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654672384:192.168.201.107@tcp:250:kworker.0 00000800:00000100:2.0:1787756182.614182:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756182.614222:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756182/real 1787756182] req@ffff8c05a9f78a80 x1874588654672384/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756182.614267:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588654672384/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756182.614283:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756182.614289:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756182.614297:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654672384:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787756182.614308:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654672512:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756182.614335:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756182.614349:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756182/real 1787756182] req@ffff8c05a9f7aa00 x1874588654672512/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756182.614359:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588654672512/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756182.614366:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756182.614369:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756182.614371:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654672512:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756182.614389:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654672640:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756182.614402:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756182.614412:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756182/real 1787756182] req@ffff8c05a9f79f80 x1874588654672640/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756182.614424:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588654672640/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756182.614440:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756182.614444:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756182.614447:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654672640:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756182.614452:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654672768:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756182.614469:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756182.614477:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756182/real 1787756182] req@ffff8c05a9f78e00 x1874588654672768/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756182.614484:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588654672768/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756182.614490:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756182.614492:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756182.614494:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654672768:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756182.614497:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654672896:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756182.614507:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756182.614512:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756182/real 1787756182] req@ffff8c05a9f79180 x1874588654672896/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756182.614519:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588654672896/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756182.614524:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756182.614526:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756182.614529:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654672896:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756182.614532:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654673024:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756182.614542:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756182.614547:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756182/real 1787756182] req@ffff8c05a9f7b800 x1874588654673024/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756182.614554:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588654673024/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756182.614558:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756182.614560:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756182.614562:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654673024:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756182.614565:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654673152:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756182.614574:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756182.614579:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756182/real 1787756182] req@ffff8c05a9f79500 x1874588654673152/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756182.614585:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588654673152/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756182.614590:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756182.614592:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756182.614594:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654673152:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756182.614597:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654673280:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756182.614607:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756182.614617:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756182/real 1787756182] req@ffff8c05a9f7b100 x1874588654673280/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756182.614629:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588654673280/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756182.614636:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756182.614639:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756182.614642:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654673280:192.168.201.107@tcp:8:kworker.0 00010000:00010000:2.0:1787756183.190054:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756185.238053:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787756187.734104:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654673408:192.168.201.107@tcp:250:kworker.0 00000800:00000100:2.0:1787756187.734180:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756187.734227:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756187/real 1787756187] req@ffff8c05a9f7bb80 x1874588654673408/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756187.734250:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588654673408/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756187.734267:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756187.734272:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756187.734280:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654673408:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787756187.734305:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654673536:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756187.734343:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756187.734367:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756187/real 1787756187] req@ffff8c05a9f7ad80 x1874588654673536/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756187.734417:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588654673536/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756187.734425:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756187.734428:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756187.734431:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654673536:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756187.734437:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654673664:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756187.734454:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756187.734462:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756187/real 1787756187] req@ffff8c05a9f7b100 x1874588654673664/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756187.734478:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588654673664/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756187.734483:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756187.734485:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756187.734488:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654673664:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756187.734492:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654673792:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756187.734502:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756187.734508:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756187/real 1787756187] req@ffff8c05a9f79500 x1874588654673792/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756187.734515:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588654673792/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756187.734524:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756187.734526:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756187.734529:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654673792:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756187.734532:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654673920:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756187.734542:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756187.734547:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756187/real 1787756187] req@ffff8c05a9f7b800 x1874588654673920/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756187.734554:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588654673920/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756187.734560:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756187.734561:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756187.734563:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654673920:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756187.734566:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654674048:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756187.734576:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756187.734582:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756187/real 1787756187] req@ffff8c05a9f79180 x1874588654674048/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756187.734589:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588654674048/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756187.734594:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756187.734595:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756187.734598:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654674048:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756187.734610:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654674176:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756187.734641:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756187.734660:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756187/real 1787756187] req@ffff8c05a9f78e00 x1874588654674176/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756187.734668:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588654674176/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756187.734673:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756187.734675:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756187.734677:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654674176:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756187.734681:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654674304:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756187.734692:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756187.734698:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756187/real 1787756187] req@ffff8c05a9f79f80 x1874588654674304/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756187.734706:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588654674304/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756187.734710:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756187.734712:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756187.734714:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654674304:192.168.201.107@tcp:8:kworker.0 00010000:00010000:2.0:1787756189.334044:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756190.358072:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756191.382015:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787756192.854189:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654674432:192.168.201.107@tcp:250:kworker.0 00000800:00000100:2.0:1787756192.854273:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756192.854324:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756192/real 1787756192] req@ffff8c05b2d9a300 x1874588654674432/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756192.854352:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588654674432/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756192.854379:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756192.854388:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756192.854395:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654674432:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787756192.854409:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654674560:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756192.854436:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756192.854448:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756192/real 1787756192] req@ffff8c05b2d99880 x1874588654674560/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756192.854458:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654674560/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756192.854464:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756192.854467:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756192.854471:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654674560:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756192.854489:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654674688:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756192.854501:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756192.854507:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756192/real 1787756192] req@ffff8c05b2d99f80 x1874588654674688/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756192.854515:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654674688/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756192.854521:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756192.854523:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756192.854525:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654674688:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756192.854545:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654674816:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756192.854561:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756192.854573:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756192/real 1787756192] req@ffff8c05b2d9b480 x1874588654674816/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756192.854586:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654674816/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756192.854594:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756192.854598:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756192.854601:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654674816:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756192.854608:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654674944:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756192.854625:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756192.854633:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756192/real 1787756192] req@ffff8c05b2d9aa00 x1874588654674944/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756192.854641:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588654674944/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756192.854650:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756192.854653:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756192.854655:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654674944:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756192.854658:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654675072:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756192.854669:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756192.854675:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756192/real 1787756192] req@ffff8c05b2d9b100 x1874588654675072/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756192.854682:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654675072/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756192.854687:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756192.854689:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756192.854690:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654675072:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756192.854693:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654675200:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756192.854703:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756192.854708:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756192/real 1787756192] req@ffff8c05b2d9a680 x1874588654675200/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756192.854715:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a680 x1874588654675200/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756192.854720:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756192.854722:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756192.854724:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654675200:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756192.854728:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654675328:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756192.854737:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756192.854742:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756192/real 1787756192] req@ffff8c05b2d98000 x1874588654675328/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756192.854749:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654675328/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756192.854756:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756192.854759:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756192.854762:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654675328:192.168.201.107@tcp:8:kworker.0 00010000:00010000:2.0:1787756193.430085:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756197.526032:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756198.550014:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756201.622127:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756205.718161:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756206.742128:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756209.814108:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756213.910052:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756214.934017:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756218.006039:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756222.102042:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756223.126065:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756226.198106:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756230.294143:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756231.318068:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756234.390038:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756238.486019:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756239.510043:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756242.582064:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756246.678129:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756247.702052:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756249.174182:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756249.174212:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787756249.174218:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756249.174231:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9305 00000100:00080000:2.0:1787756249.174243:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756249.174250:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756249.174291:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588654685696/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756249.174326:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756249.174330:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756249.174332:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756249.174336:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9305 00000100:00080000:2.0:1787756249.174339:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756249.174343:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756249.174352:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756249.174363:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588654685824/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756249.174385:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756249.174389:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756249.174391:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756249.174396:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9305 00000100:00080000:2.0:1787756249.174399:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756249.174403:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756249.174408:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756249.174422:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588654685952/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756249.174434:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756249.174448:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756249.174450:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756249.174453:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9305 00000100:00080000:2.0:1787756249.174456:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756249.174461:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756249.174465:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756249.174498:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588654686080/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756249.174515:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756249.174520:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756249.174523:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756249.174527:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9305 00000100:00080000:2.0:1787756249.174532:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756249.174616:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756249.174625:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756249.174655:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588654686208/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756249.174670:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756249.174675:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756249.174678:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756249.174684:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9305 00000100:00080000:2.0:1787756249.174693:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756249.174707:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756249.174720:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756249.174738:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588654686336/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756249.174755:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756249.174763:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787756249.174769:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756249.174778:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9305 00000100:00080000:2.0:1787756249.174782:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756249.174796:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756249.174821:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756249.174843:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588654686464/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756249.174855:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756249.174874:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756249.174877:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756249.174883:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9305 00000100:00080000:2.0:1787756249.174889:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756249.174895:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756249.174924:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756249.174977:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588654686592/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787756250.774126:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756254.293973:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756254.293988:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787756254.293994:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756254.294007:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9310 00000100:00080000:2.0:1787756254.294022:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756254.294031:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756254.294094:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588654686720/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756254.294128:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756254.294132:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756254.294134:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756254.294139:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9310 00000100:00080000:2.0:1787756254.294143:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756254.294148:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756254.294160:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756254.294175:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71f80 x1874588654686848/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756254.294190:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756254.294199:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756254.294201:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756254.294206:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9310 00000100:00080000:2.0:1787756254.294209:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756254.294213:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756254.294218:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756254.294230:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588654686976/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756254.294242:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756254.294246:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756254.294248:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756254.294252:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9310 00000100:00080000:2.0:1787756254.294255:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756254.294260:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756254.294271:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756254.294284:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588654687104/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756254.294295:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756254.294299:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756254.294301:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756254.294305:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9310 00000100:00080000:2.0:1787756254.294310:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756254.294325:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756254.294331:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756254.294341:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588654687232/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756254.294352:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756254.294358:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756254.294361:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756254.294370:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9310 00000100:00080000:2.0:1787756254.294381:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756254.294389:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756254.294399:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756254.294418:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588654687360/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756254.294430:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756254.294434:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787756254.294437:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756254.294442:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9310 00000100:00080000:2.0:1787756254.294461:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756254.294468:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756254.294473:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756254.294506:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588654687488/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756254.294532:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756254.294539:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756254.294544:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756254.294558:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9310 00000100:00080000:2.0:1787756254.294566:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756254.294575:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756254.294583:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756254.294599:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588654687616/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787756254.870160:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756255.894079:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756258.966057:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756263.062013:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756264.086059:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756264.534029:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756264.534040:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787756264.534045:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756264.534055:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9320 00000100:00080000:2.0:1787756264.534065:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756264.534074:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756264.534109:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73800 x1874588654688768/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756264.534135:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756264.534138:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756264.534140:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756264.534143:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9320 00000100:00080000:2.0:1787756264.534146:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756264.534150:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756264.534157:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756264.534168:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588654688896/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756264.534176:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756264.534178:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756264.534179:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756264.534182:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9320 00000100:00080000:2.0:1787756264.534184:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756264.534186:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756264.534189:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756264.534198:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588654689024/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756264.534205:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756264.534208:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756264.534210:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756264.534213:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9320 00000100:00080000:2.0:1787756264.534215:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756264.534220:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756264.534224:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756264.534236:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588654689152/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756264.534250:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756264.534256:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756264.534260:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756264.534293:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9320 00000100:00080000:2.0:1787756264.534309:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756264.534314:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756264.534320:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756264.534337:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588654689280/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756264.534349:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756264.534352:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756264.534354:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756264.534359:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9320 00000100:00080000:2.0:1787756264.534366:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756264.534371:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756264.534376:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756264.534390:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588654689408/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756264.534420:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756264.534427:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787756264.534429:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756264.534436:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9320 00000100:00080000:2.0:1787756264.534443:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756264.534447:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756264.534453:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756264.534469:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588654689536/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756264.534481:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756264.534485:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756264.534491:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756264.534505:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9320 00000100:00080000:2.0:1787756264.534520:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756264.534531:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756264.534540:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756264.534561:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588654689664/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787756267.158137:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756271.254031:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756272.278037:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756275.350042:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787756277.846140:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.107:988 00000400:00000100:2.0:1787756277.846177:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.107@tcp at host 192.168.201.107 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787756277.846191:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787756277.846220:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787756277.846227:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787756277.846234:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787756277.846242:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787756277.846250:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787756277.846257:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787756277.846264:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00010000:00010000:2.0:1787756279.446135:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756280.470110:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756283.542068:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756285.014026:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756285.014035:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787756285.014040:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756285.014048:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9341 00000100:00080000:2.0:1787756285.014058:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756285.014064:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756285.014117:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71f80 x1874588654692864/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756285.014146:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756285.014149:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756285.014151:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756285.014170:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9341 00000100:00080000:2.0:1787756285.014174:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756285.014178:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756285.014187:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756285.014200:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588654692992/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756285.014211:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756285.014214:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756285.014215:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756285.014221:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9341 00000100:00080000:2.0:1787756285.014225:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756285.014249:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756285.014256:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756285.014267:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588654693120/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756285.014275:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756285.014278:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756285.014279:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756285.014282:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9341 00000100:00080000:2.0:1787756285.014285:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756285.014288:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756285.014292:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756285.014300:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588654693248/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756285.014307:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756285.014310:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756285.014311:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756285.014313:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9341 00000100:00080000:2.0:1787756285.014318:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756285.014323:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756285.014328:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756285.014338:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588654693376/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756285.014362:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756285.014364:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756285.014366:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756285.014370:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9341 00000100:00080000:2.0:1787756285.014373:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756285.014379:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756285.014393:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756285.014407:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588654693504/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756285.014418:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756285.014426:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787756285.014427:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756285.014432:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9341 00000100:00080000:2.0:1787756285.014438:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756285.014445:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756285.014452:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756285.014463:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588654693632/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756285.014469:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756285.014472:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756285.014474:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756285.014481:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9341 00000100:00080000:2.0:1787756285.014487:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756285.014491:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756285.014495:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756285.014504:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588654693760/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787756287.638220:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756288.662194:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756290.134002:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756290.134016:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787756290.134022:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756290.134036:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9346 00000100:00080000:2.0:1787756290.134051:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756290.134060:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756290.134152:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588654693888/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756290.134192:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756290.134197:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756290.134199:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756290.134204:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9346 00000100:00080000:2.0:1787756290.134208:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756290.134212:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756290.134224:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756290.134239:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588654694016/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756290.134258:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756290.134262:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756290.134265:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756290.134272:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9346 00000100:00080000:2.0:1787756290.134276:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756290.134280:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756290.134286:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756290.134320:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588654694144/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756290.134336:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756290.134340:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756290.134342:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756290.134346:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9346 00000100:00080000:2.0:1787756290.134350:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756290.134355:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756290.134363:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756290.134375:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588654694272/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756290.134402:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756290.134406:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756290.134409:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756290.134413:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9346 00000100:00080000:2.0:1787756290.134418:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756290.134424:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756290.134429:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756290.134439:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588654694400/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756290.134510:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756290.134516:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756290.134518:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756290.134522:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9346 00000100:00080000:2.0:1787756290.134526:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756290.134533:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756290.134542:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756290.134562:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588654694528/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756290.134594:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756290.134602:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787756290.134605:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756290.134611:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9346 00000100:00080000:2.0:1787756290.134621:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756290.134626:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756290.134633:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756290.134647:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588654694656/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756290.134665:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756290.134670:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756290.134673:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756290.134685:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9346 00000100:00080000:2.0:1787756290.134694:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756290.134699:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756290.134704:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756290.134721:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588654694784/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787756291.734105:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756295.254006:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756295.254020:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787756295.254026:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756295.254039:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9351 00000100:00080000:2.0:1787756295.254053:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756295.254063:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756295.254130:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588654694912/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756295.254165:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756295.254169:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756295.254171:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756295.254175:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9351 00000100:00080000:2.0:1787756295.254179:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756295.254184:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756295.254196:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756295.254208:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71f80 x1874588654695040/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756295.254219:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756295.254234:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756295.254236:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756295.254240:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9351 00000100:00080000:2.0:1787756295.254244:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756295.254248:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756295.254252:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756295.254261:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588654695168/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756295.254273:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756295.254278:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756295.254282:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756295.254292:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9351 00000100:00080000:2.0:1787756295.254296:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756295.254299:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756295.254304:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756295.254323:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588654695296/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756295.254338:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756295.254345:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756295.254348:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756295.254353:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9351 00000100:00080000:2.0:1787756295.254360:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756295.254367:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756295.254375:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756295.254401:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588654695424/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756295.254418:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756295.254424:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756295.254428:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756295.254434:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9351 00000100:00080000:2.0:1787756295.254440:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756295.254444:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756295.254452:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756295.254472:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588654695552/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756295.254544:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756295.254554:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787756295.254557:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756295.254576:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9351 00000100:00080000:2.0:1787756295.254591:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756295.254596:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756295.254604:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756295.254625:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588654695680/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756295.254660:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756295.254667:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756295.254670:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756295.254683:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9351 00000100:00080000:2.0:1787756295.254695:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756295.254711:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756295.254750:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756295.254778:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588654695808/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787756295.830213:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756296.854119:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756299.926034:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756304.022031:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756305.046318:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756308.118228:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756312.214103:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756313.238064:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756315.734064:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756315.734076:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787756315.734080:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756315.734091:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9372 00000100:00080000:2.0:1787756315.734103:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756315.734125:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756315.734167:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588654699008/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756315.734204:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756315.734208:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756315.734210:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756315.734214:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9372 00000100:00080000:2.0:1787756315.734218:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756315.734222:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756315.734231:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756315.734242:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73800 x1874588654699136/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756315.734252:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756315.734256:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756315.734257:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756315.734261:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9372 00000100:00080000:2.0:1787756315.734263:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756315.734266:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756315.734270:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756315.734292:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588654699264/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756315.734303:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756315.734306:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756315.734308:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756315.734313:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9372 00000100:00080000:2.0:1787756315.734317:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756315.734360:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756315.734382:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756315.734399:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588654699392/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756315.734421:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756315.734425:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756315.734428:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756315.734433:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9372 00000100:00080000:2.0:1787756315.734439:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756315.734446:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756315.734456:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756315.734474:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588654699520/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756315.734486:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756315.734489:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756315.734490:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756315.734494:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9372 00000100:00080000:2.0:1787756315.734497:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756315.734501:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756315.734507:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756315.734526:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588654699648/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756315.734548:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756315.734555:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787756315.734560:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756315.734570:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9372 00000100:00080000:2.0:1787756315.734576:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756315.734584:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756315.734591:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756315.734605:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588654699776/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756315.734621:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756315.734624:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756315.734629:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756315.734638:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9372 00000100:00080000:2.0:1787756315.734645:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756315.734650:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756315.734664:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756315.734682:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588654699904/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787756316.310173:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756320.406243:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756321.430048:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756324.502198:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756328.598217:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756329.622293:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756332.694142:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756336.790149:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756337.814183:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756340.886139:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787756341.334369:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654704128:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787756341.334565:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654704256:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756341.334635:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654704384:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756341.334687:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654704512:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756341.334738:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654704640:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756341.334779:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654704768:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756341.334825:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654704896:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756341.335103:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654705024:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756344.406603:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756341/real 1787756344] req@ffff8c05b2d98700 x1874588654704256/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756344.406695:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654704256/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756344.406714:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756344.406723:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756344.406729:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654704256:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756344.406741:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756341/real 1787756344] req@ffff8c05b2d98a80 x1874588654704384/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756344.406759:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98a80 x1874588654704384/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756344.406770:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756344.406776:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756344.406781:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654704384:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756344.406790:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756341/real 1787756344] req@ffff8c05b2d9a680 x1874588654704512/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756344.406822:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a680 x1874588654704512/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756344.406833:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756344.406838:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756344.406843:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654704512:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756344.406853:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756341/real 1787756344] req@ffff8c05b2d98380 x1874588654704128/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756344.406920:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654704128/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756344.406931:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756344.406936:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756344.406940:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654704128:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787756344.406982:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756341/real 1787756344] req@ffff8c05b2d9a300 x1874588654705024/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756344.407002:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588654705024/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756344.407013:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756344.407018:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756344.407022:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654705024:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756344.407032:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756341/real 1787756344] req@ffff8c05b2d9b480 x1874588654704896/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756344.407049:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654704896/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756344.407059:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756344.407064:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756344.407068:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654704896:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756344.407077:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756341/real 1787756344] req@ffff8c05b2d9bb80 x1874588654704640/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756344.407092:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588654704640/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756344.407103:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756344.407107:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756344.407111:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654704640:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756344.407120:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756341/real 1787756344] req@ffff8c05b2d98000 x1874588654704768/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756344.407137:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654704768/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756344.407148:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756344.407153:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756344.407158:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654704768:192.168.201.107@tcp:38:kworker.0 00010000:00010000:2.0:1787756344.982162:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756346.006070:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787756346.454272:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654705152:192.168.201.107@tcp:250:kworker.0 00000800:00000100:2.0:1787756346.454420:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756346.454523:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756346/real 1787756346] req@ffff8c05a9f7ad80 x1874588654705152/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756346.454583:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588654705152/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756346.454608:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756346.454623:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756346.454636:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654705152:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787756346.454698:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654705280:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756346.454753:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756346.454788:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756346/real 1787756346] req@ffff8c05a9f7bb80 x1874588654705280/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756346.454856:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588654705280/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756346.454872:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756346.454984:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756346.455002:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654705280:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756346.455025:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654705408:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756346.455083:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756346.455111:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756346/real 1787756346] req@ffff8c05a9f7b100 x1874588654705408/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756346.455125:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588654705408/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756346.455134:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756346.455138:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756346.455142:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654705408:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756346.455151:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654705536:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756346.455176:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756346.455191:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756346/real 1787756346] req@ffff8c05a9f79500 x1874588654705536/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756346.455205:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588654705536/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756346.455219:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756346.455224:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756346.455231:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654705536:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756346.455243:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654705664:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756346.455271:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756346.455305:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756346/real 1787756346] req@ffff8c05a9f7b800 x1874588654705664/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756346.455320:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588654705664/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756346.455329:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756346.455333:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756346.455337:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654705664:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756346.455364:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654705792:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756346.455402:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756346.455416:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756346/real 1787756346] req@ffff8c05a9f79180 x1874588654705792/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756346.455430:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588654705792/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756346.455438:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756346.455442:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756346.455445:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654705792:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756346.455452:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654705920:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756346.455475:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756346.455489:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756346/real 1787756346] req@ffff8c05a9f78e00 x1874588654705920/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756346.455501:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588654705920/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756346.455509:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756346.455512:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756346.455515:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654705920:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756346.455537:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654706048:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756346.455561:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756346.455573:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756346/real 1787756346] req@ffff8c05a9f79f80 x1874588654706048/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756346.455585:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588654706048/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756346.455594:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756346.455597:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756346.455600:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654706048:192.168.201.107@tcp:8:kworker.0 00010000:00010000:2.0:1787756349.078193:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787756351.574212:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654706176:192.168.201.107@tcp:250:kworker.0 00000800:00000100:2.0:1787756351.574319:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756351.574410:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756351/real 1787756351] req@ffff8c05a9f78000 x1874588654706176/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756351.574463:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78000 x1874588654706176/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756351.574494:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756351.574507:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756351.574519:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654706176:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787756351.574543:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654706304:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756351.574617:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756351.574660:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756351/real 1787756351] req@ffff8c05a9f78700 x1874588654706304/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756351.574682:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588654706304/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756351.574693:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756351.574698:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756351.574702:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654706304:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756351.574764:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654706432:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756351.574798:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756351.574814:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756351/real 1787756351] req@ffff8c05a9f7a680 x1874588654706432/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756351.574830:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588654706432/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756351.574841:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756351.574845:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756351.574849:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654706432:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756351.574858:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654706560:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756351.574922:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756351.574938:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756351/real 1787756351] req@ffff8c05a9f78a80 x1874588654706560/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756351.574954:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588654706560/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756351.574965:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756351.574968:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756351.574973:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654706560:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756351.574980:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654706688:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756351.575003:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756351.575015:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756351/real 1787756351] req@ffff8c05a9f7b480 x1874588654706688/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756351.575029:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588654706688/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756351.575039:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756351.575042:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756351.575046:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654706688:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756351.575053:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654706816:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756351.575072:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756351.575083:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756351/real 1787756351] req@ffff8c05a9f79c00 x1874588654706816/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756351.575098:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79c00 x1874588654706816/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756351.575107:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756351.575110:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756351.575114:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654706816:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756351.575120:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654706944:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756351.575139:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756351.575149:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756351/real 1787756351] req@ffff8c05a9f7aa00 x1874588654706944/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756351.575162:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588654706944/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756351.575172:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756351.575175:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756351.575180:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654706944:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756351.575186:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654707072:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756351.575203:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756351.575214:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756351/real 1787756351] req@ffff8c05a9f79880 x1874588654707072/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756351.575226:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588654707072/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756351.575244:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756351.575248:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756351.575252:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654707072:192.168.201.107@tcp:8:kworker.0 00010000:00010000:2.0:1787756353.174129:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756354.198137:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756356.694116:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756356.694134:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787756356.694141:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756356.694157:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9413 00000100:00080000:2.0:1787756356.694173:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756356.694186:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756356.694253:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71f80 x1874588654707200/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756356.694302:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756356.694307:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756356.694310:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756356.694316:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9413 00000100:00080000:2.0:1787756356.694321:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756356.694325:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756356.694334:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756356.694347:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588654707328/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756356.694357:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756356.694361:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756356.694363:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756356.694366:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9413 00000100:00080000:2.0:1787756356.694369:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756356.694372:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756356.694376:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756356.694389:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588654707456/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756356.694418:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756356.694421:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756356.694423:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756356.694426:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9413 00000100:00080000:2.0:1787756356.694428:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756356.694431:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756356.694435:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756356.694445:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588654707584/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756356.694453:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756356.694455:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756356.694457:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756356.694460:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9413 00000100:00080000:2.0:1787756356.694463:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756356.694466:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756356.694471:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756356.694482:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588654707712/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756356.694491:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756356.694494:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756356.694495:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756356.694499:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9413 00000100:00080000:2.0:1787756356.694502:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756356.694505:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756356.694509:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756356.694517:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588654707840/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756356.694524:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756356.694527:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787756356.694528:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756356.694532:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9413 00000100:00080000:2.0:1787756356.694534:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756356.694537:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756356.694541:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756356.694550:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588654707968/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756356.694563:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756356.694572:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756356.694577:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756356.694587:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9413 00000100:00080000:2.0:1787756356.694590:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756356.694594:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756356.694609:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756356.694624:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588654708096/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787756357.270128:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756361.366110:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756362.390111:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756365.462094:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756369.558246:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756370.582091:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756373.654096:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756377.750204:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756378.774217:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756381.846257:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756385.942162:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756386.966215:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756390.038230:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756392.534049:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756392.534060:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787756392.534065:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756392.534075:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9448 00000100:00080000:2.0:1787756392.534086:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756392.534092:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756392.534127:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588654714368/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756392.534154:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756392.534156:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756392.534158:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756392.534161:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9448 00000100:00080000:2.0:1787756392.534163:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756392.534165:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756392.534172:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756392.534182:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588654714496/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756392.534189:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756392.534192:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756392.534193:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756392.534196:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9448 00000100:00080000:2.0:1787756392.534199:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756392.534201:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756392.534206:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756392.534220:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588654714624/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756392.534228:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756392.534230:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756392.534232:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756392.534235:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9448 00000100:00080000:2.0:1787756392.534238:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756392.534241:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756392.534245:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756392.534252:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588654714752/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756392.534259:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756392.534261:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756392.534263:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756392.534264:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9448 00000100:00080000:2.0:1787756392.534267:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756392.534272:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756392.534278:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756392.534286:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588654714880/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756392.534292:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756392.534295:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756392.534297:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756392.534300:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9448 00000100:00080000:2.0:1787756392.534304:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756392.534309:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756392.534314:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756392.534330:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588654715008/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756392.534357:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756392.534362:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787756392.534365:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756392.534372:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9448 00000100:00080000:2.0:1787756392.534379:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756392.534393:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756392.534398:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756392.534412:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588654715136/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756392.534424:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756392.534431:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756392.534435:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756392.534445:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9448 00000100:00080000:2.0:1787756392.534454:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756392.534462:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756392.534473:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756392.534489:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588654715264/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787756394.134152:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756395.158150:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756398.230083:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756402.326026:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756403.350088:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756406.422170:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756410.518188:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756411.542107:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787756413.014445:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654718464:192.168.201.107@tcp:250:kworker.0 00000800:00000100:2.0:1787756413.014574:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756413.014685:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756413/real 1787756413] req@ffff8c05b2d98700 x1874588654718464/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756413.014751:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654718464/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756413.014780:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756413.014793:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756413.014804:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654718464:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787756413.014853:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654718592:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756413.014970:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756413.015002:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756413/real 1787756413] req@ffff8c05b2d98a80 x1874588654718592/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756413.015041:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98a80 x1874588654718592/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756413.015078:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756413.015087:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756413.015094:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654718592:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756413.015104:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654718720:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756413.015142:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756413.015160:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756413/real 1787756413] req@ffff8c05b2d9a680 x1874588654718720/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756413.015176:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a680 x1874588654718720/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756413.015187:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756413.015191:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756413.015197:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654718720:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756413.015206:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654718848:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756413.015236:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756413.015253:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756413/real 1787756413] req@ffff8c05b2d98380 x1874588654718848/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756413.015269:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654718848/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756413.015280:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756413.015285:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756413.015289:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654718848:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756413.015298:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654718976:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756413.015325:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756413.015340:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756413/real 1787756413] req@ffff8c05b2d9a300 x1874588654718976/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756413.015367:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588654718976/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756413.015379:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756413.015383:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756413.015389:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654718976:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756413.015397:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654719104:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756413.015425:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756413.015439:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756413/real 1787756413] req@ffff8c05b2d9b480 x1874588654719104/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756413.015456:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654719104/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756413.015466:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756413.015471:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756413.015475:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654719104:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756413.015507:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654719232:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756413.015535:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756413.015551:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756413/real 1787756413] req@ffff8c05b2d9b100 x1874588654719232/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756413.015567:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654719232/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756413.015586:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756413.015590:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756413.015595:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654719232:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756413.015603:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654719360:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756413.015629:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756413.015644:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756413/real 1787756413] req@ffff8c05b2d9aa00 x1874588654719360/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756413.015680:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588654719360/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756413.015691:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756413.015696:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756413.015700:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654719360:192.168.201.107@tcp:8:kworker.0 00010000:00010000:2.0:1787756414.614073:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787756418.134238:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654719488:192.168.201.107@tcp:250:kworker.0 00000800:00000100:2.0:1787756418.134330:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756418.134405:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756418/real 1787756418] req@ffff8c05a9f7ad80 x1874588654719488/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756418.134430:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588654719488/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756418.134459:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756418.134468:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756418.134479:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654719488:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787756418.134515:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654719616:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756418.134576:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756418.134597:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756418/real 1787756418] req@ffff8c05a9f7bb80 x1874588654719616/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756418.134617:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588654719616/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756418.134638:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756418.134642:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756418.134646:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654719616:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756418.134653:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654719744:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756418.134674:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756418.134683:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756418/real 1787756418] req@ffff8c05a9f7b100 x1874588654719744/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756418.134693:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588654719744/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756418.134700:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756418.134702:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756418.134705:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654719744:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756418.134709:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654719872:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756418.134723:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756418.134731:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756418/real 1787756418] req@ffff8c05a9f79500 x1874588654719872/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756418.134740:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588654719872/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756418.134746:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756418.134748:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756418.134751:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654719872:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756418.134755:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654720000:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756418.134767:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756418.134774:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756418/real 1787756418] req@ffff8c05a9f7b800 x1874588654720000/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756418.134784:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588654720000/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756418.134790:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756418.134792:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756418.134794:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654720000:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756418.134798:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654720128:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756418.134809:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756418.134816:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756418/real 1787756418] req@ffff8c05a9f79180 x1874588654720128/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756418.134823:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588654720128/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756418.134830:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756418.134833:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756418.134836:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654720128:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756418.134850:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654720256:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756418.134862:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756418.134869:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756418/real 1787756418] req@ffff8c05a9f78e00 x1874588654720256/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756418.134920:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588654720256/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756418.134927:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756418.134929:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756418.134932:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654720256:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756418.134937:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654720384:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756418.134953:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756418.134961:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756418/real 1787756418] req@ffff8c05a9f79f80 x1874588654720384/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756418.134970:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588654720384/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756418.134977:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756418.134981:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756418.134986:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654720384:192.168.201.107@tcp:8:kworker.0 00010000:00010000:2.0:1787756418.710126:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756419.734161:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756422.806117:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787756423.254549:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654720512:192.168.201.107@tcp:250:kworker.0 00000800:00000100:2.0:1787756423.254665:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756423.254751:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756423/real 1787756423] req@ffff8c05b2d9bb80 x1874588654720512/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756423.254822:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588654720512/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756423.254929:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756423.254938:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756423.254965:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654720512:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787756423.255027:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654720640:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756423.255089:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756423.255125:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756423/real 1787756423] req@ffff8c05b2d98000 x1874588654720640/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756423.255172:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654720640/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756423.255217:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756423.255237:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756423.255251:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654720640:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756423.255285:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654720768:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756423.255348:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756423.255380:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756423/real 1787756423] req@ffff8c05b2d99880 x1874588654720768/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756423.255425:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654720768/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756423.255462:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756423.255501:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756423.255511:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654720768:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756423.255530:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654720896:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756423.255587:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756423.255617:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756423/real 1787756423] req@ffff8c05b2d99f80 x1874588654720896/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756423.255656:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654720896/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756423.255683:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756423.255690:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756423.255697:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654720896:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756423.255727:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654721024:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756423.255758:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756423.255776:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756423/real 1787756423] req@ffff8c05b2d9bb80 x1874588654721024/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756423.255801:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588654721024/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756423.255817:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756423.255823:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756423.255830:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654721024:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756423.255841:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654721152:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756423.255869:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756423.255923:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756423/real 1787756423] req@ffff8c05b2d9aa00 x1874588654721152/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756423.255947:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588654721152/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756423.255963:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756423.255970:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756423.255985:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654721152:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756423.255997:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654721280:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756423.256029:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756423.256048:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756423/real 1787756423] req@ffff8c05b2d9b100 x1874588654721280/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756423.256071:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654721280/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756423.256101:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756423.256107:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756423.256114:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654721280:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756423.256124:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654721408:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756423.256151:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756423.256168:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756423/real 1787756423] req@ffff8c05b2d9b480 x1874588654721408/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756423.256190:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654721408/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756423.256207:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756423.256212:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756423.256219:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654721408:192.168.201.107@tcp:8:kworker.0 00010000:00010000:2.0:1787756426.902087:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756427.926213:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756428.374055:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756428.374074:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787756428.374081:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756428.374099:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9484 00000100:00080000:2.0:1787756428.374117:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756428.374129:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756428.374177:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588654721536/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756428.374233:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756428.374238:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756428.374241:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756428.374246:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9484 00000100:00080000:2.0:1787756428.374251:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756428.374255:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756428.374266:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756428.374282:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71f80 x1874588654721664/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756428.374311:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756428.374316:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756428.374318:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756428.374324:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9484 00000100:00080000:2.0:1787756428.374328:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756428.374334:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756428.374341:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756428.374359:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588654721792/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756428.374403:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756428.374408:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756428.374410:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756428.374415:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9484 00000100:00080000:2.0:1787756428.374420:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756428.374425:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756428.374432:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756428.374446:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588654721920/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756428.374455:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756428.374458:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756428.374460:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756428.374466:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9484 00000100:00080000:2.0:1787756428.374473:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756428.374480:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756428.374487:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756428.374505:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588654722048/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756428.374519:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756428.374524:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756428.374528:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756428.374538:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9484 00000100:00080000:2.0:1787756428.374550:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756428.374558:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756428.374567:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756428.374590:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588654722176/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756428.374626:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756428.374634:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787756428.374636:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756428.374642:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9484 00000100:00080000:2.0:1787756428.374647:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756428.374651:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756428.374656:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756428.374674:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588654722304/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756428.374716:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756428.374731:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756428.374738:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756428.374749:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9484 00000100:00080000:2.0:1787756428.374759:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756428.374781:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756428.374788:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756428.374806:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588654722432/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787756430.998152:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756435.094192:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756436.118196:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756438.614103:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756438.614121:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787756438.614142:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756438.614158:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9495 00000100:00080000:2.0:1787756438.614178:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756438.614188:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756438.614273:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73800 x1874588654723584/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756438.614325:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756438.614332:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756438.614336:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756438.614344:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9495 00000100:00080000:2.0:1787756438.614352:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756438.614361:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756438.614380:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756438.614409:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588654723712/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756438.614426:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756438.614431:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756438.614434:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756438.614452:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9495 00000100:00080000:2.0:1787756438.614459:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756438.614465:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756438.614472:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756438.614491:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588654723840/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756438.614531:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756438.614556:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756438.614561:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756438.614574:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9495 00000100:00080000:2.0:1787756438.614583:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756438.614592:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756438.614605:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756438.614644:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588654723968/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756438.614691:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756438.614722:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756438.614733:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756438.614747:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9495 00000100:00080000:2.0:1787756438.614754:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756438.614771:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756438.614781:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756438.614806:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588654724096/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756438.614823:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756438.614828:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756438.614831:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756438.614852:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9495 00000100:00080000:2.0:1787756438.614859:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756438.614870:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756438.614877:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756438.614901:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588654724224/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756438.614961:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756438.614965:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787756438.614968:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756438.614974:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9495 00000100:00080000:2.0:1787756438.614978:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756438.614984:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756438.614991:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756438.615010:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588654724352/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756438.615024:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756438.615031:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756438.615034:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756438.615040:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9495 00000100:00080000:2.0:1787756438.615043:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756438.615048:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756438.615057:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756438.615080:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588654724480/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787756439.190161:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756443.286084:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756444.309996:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756447.382219:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787756448.854349:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654725632:192.168.201.107@tcp:250:kworker.0 00000800:00000100:2.0:1787756448.854459:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756448.854541:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756448/real 1787756448] req@ffff8c05a9f78380 x1874588654725632/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756448.854570:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588654725632/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756448.854585:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756448.854592:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756448.854598:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654725632:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787756448.854631:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654725760:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756448.854671:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756448.854691:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756448/real 1787756448] req@ffff8c05a9f7a300 x1874588654725760/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756448.854706:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588654725760/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756448.854749:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756448.854756:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756448.854761:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654725760:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756448.854772:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654725888:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756448.854802:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756448.854821:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756448/real 1787756448] req@ffff8c05a9f79880 x1874588654725888/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756448.854836:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588654725888/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756448.854847:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756448.854852:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756448.854856:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654725888:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756448.855252:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654726016:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756448.855332:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756448.855366:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756448/real 1787756448] req@ffff8c05a9f7aa00 x1874588654726016/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756448.855421:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588654726016/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756448.855453:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756448.855463:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756448.855473:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654726016:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756448.855520:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654726144:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756448.855566:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756448.855595:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756448/real 1787756448] req@ffff8c05a9f79c00 x1874588654726144/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756448.855617:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79c00 x1874588654726144/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756448.855628:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756448.855635:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756448.855640:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654726144:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756448.855648:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654726272:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756448.855667:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756448.855678:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756448/real 1787756448] req@ffff8c05a9f7b480 x1874588654726272/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756448.855692:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588654726272/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756448.855700:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756448.855703:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756448.855707:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654726272:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756448.855713:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654726400:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756448.855730:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756448.855740:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756448/real 1787756448] req@ffff8c05a9f78a80 x1874588654726400/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756448.855752:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588654726400/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756448.855768:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756448.855772:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756448.855776:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654726400:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756448.855792:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654726528:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756448.855810:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756448.855820:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756448/real 1787756448] req@ffff8c05a9f7a680 x1874588654726528/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756448.855833:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588654726528/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756448.855841:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756448.855845:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756448.855848:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654726528:192.168.201.107@tcp:8:kworker.0 00010000:00010000:2.0:1787756451.478169:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756452.502009:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787756453.974272:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654726656:192.168.201.107@tcp:250:kworker.0 00000800:00000100:2.0:1787756453.974374:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756453.974428:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756453/real 1787756453] req@ffff8c05b2d9bb80 x1874588654726656/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756453.974454:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588654726656/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756453.974467:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756453.974475:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756453.974486:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654726656:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787756453.974505:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654726784:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756453.974542:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756453.974558:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756453/real 1787756453] req@ffff8c05b2d99f80 x1874588654726784/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756453.974583:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654726784/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756453.974600:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756453.974607:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756453.974615:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654726784:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756453.974639:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654726912:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756453.974655:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756453.974662:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756453/real 1787756453] req@ffff8c05b2d99880 x1874588654726912/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756453.974670:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654726912/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756453.974675:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756453.974677:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756453.974679:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654726912:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756453.974682:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654727040:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756453.974693:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756453.974716:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756453/real 1787756453] req@ffff8c05b2d98000 x1874588654727040/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756453.974724:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654727040/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756453.974730:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756453.974732:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756453.974734:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654727040:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756453.974737:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654727168:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756453.974748:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756453.974755:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756453/real 1787756453] req@ffff8c05b2d9aa00 x1874588654727168/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756453.974762:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588654727168/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756453.974767:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756453.974769:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756453.974771:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654727168:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756453.974774:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654727296:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756453.974785:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756453.974790:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756453/real 1787756453] req@ffff8c05b2d9b100 x1874588654727296/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756453.974812:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654727296/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756453.974817:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756453.974819:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756453.974821:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654727296:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756453.974825:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654727424:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756453.974835:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756453.974840:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756453/real 1787756453] req@ffff8c05b2d9b480 x1874588654727424/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756453.974847:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654727424/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756453.974870:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756453.974872:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756453.974875:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654727424:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756453.974881:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654727552:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756453.974897:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756453.974906:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756453/real 1787756453] req@ffff8c05b2d98700 x1874588654727552/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756453.974916:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654727552/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756453.974978:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756453.974981:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756453.974983:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654727552:192.168.201.107@tcp:8:kworker.0 00010000:00010000:2.0:1787756455.574177:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756459.094088:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756459.094101:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787756459.094107:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756459.094120:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9515 00000100:00080000:2.0:1787756459.094134:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756459.094142:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756459.094188:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71f80 x1874588654727680/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756459.094237:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756459.094244:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756459.094248:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756459.094257:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9515 00000100:00080000:2.0:1787756459.094266:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756459.094274:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756459.094290:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756459.094316:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588654727808/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756459.094339:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756459.094348:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756459.094353:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756459.094361:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9515 00000100:00080000:2.0:1787756459.094369:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756459.094385:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756459.094401:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756459.094429:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588654727936/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756459.094453:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756459.094461:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756459.094469:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756459.094486:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9515 00000100:00080000:2.0:1787756459.094498:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756459.094508:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756459.094519:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756459.094552:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588654728064/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756459.094587:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756459.094605:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756459.094614:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756459.094631:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9515 00000100:00080000:2.0:1787756459.094649:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756459.094671:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756459.094687:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756459.094719:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588654728192/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756459.094757:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756459.094766:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756459.094771:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756459.094789:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9515 00000100:00080000:2.0:1787756459.094801:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756459.094816:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756459.094829:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756459.094938:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588654728320/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756459.094955:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756459.094960:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787756459.094964:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756459.094975:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9515 00000100:00080000:2.0:1787756459.094981:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756459.094986:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756459.094995:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756459.095013:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588654728448/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756459.095030:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756459.095036:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756459.095040:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756459.095046:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9515 00000100:00080000:2.0:1787756459.095056:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756459.095066:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756459.095073:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756459.095093:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588654728576/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787756459.670140:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756460.694105:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756463.766210:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756467.862149:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756468.886080:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756471.958218:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787756474.454318:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654730752:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787756474.454556:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654730880:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756474.454627:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654731008:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756474.454655:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654731136:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756474.454672:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654731264:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756474.454687:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654731392:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756474.454702:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654731520:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756474.454718:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654731648:192.168.201.107@tcp:8:kworker.0 00010000:00010000:2.0:1787756476.054221:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756477.078155:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787756477.526556:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756474/real 1787756477] req@ffff8c05a9f74700 x1874588654731008/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756477.526618:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74700 x1874588654731008/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756477.526659:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756477.526671:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756477.526679:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654731008:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756477.526692:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756474/real 1787756477] req@ffff8c05a9f76680 x1874588654731136/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756477.526712:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588654731136/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756477.526722:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756477.526727:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756477.526731:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654731136:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756477.526741:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756474/real 1787756477] req@ffff8c05a9f77b80 x1874588654731264/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756477.526755:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588654731264/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756477.526765:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756477.526769:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756477.526773:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654731264:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756477.526783:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756474/real 1787756477] req@ffff8c05a9f76d80 x1874588654731392/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756477.526798:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588654731392/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756477.526807:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756477.526812:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756477.526815:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654731392:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756477.526824:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756474/real 1787756477] req@ffff8c05a9f76a00 x1874588654731520/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756477.526838:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588654731520/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756477.526846:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756477.526851:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756477.526855:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654731520:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756477.526863:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756474/real 1787756477] req@ffff8c05a9f74380 x1874588654731648/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756477.526944:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588654731648/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756477.526956:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756477.526962:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756477.526966:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654731648:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756477.526976:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756474/real 1787756477] req@ffff8c05a9f75c00 x1874588654730752/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756477.526999:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588654730752/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756477.527010:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756477.527016:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756477.527021:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654730752:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787756477.527031:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756474/real 1787756477] req@ffff8c05a9f74a80 x1874588654730880/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756477.527048:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588654730880/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756477.527060:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756477.527066:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756477.527070:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654730880:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756479.574276:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654731776:192.168.201.107@tcp:250:kworker.0 00000800:00000100:2.0:1787756479.574440:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756479.574615:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756479/real 1787756479] req@ffff8c05a9f78000 x1874588654731776/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756479.574679:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78000 x1874588654731776/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756479.574709:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756479.574732:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756479.574749:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654731776:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787756479.574815:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654731904:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756479.574962:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756479.575009:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756479/real 1787756479] req@ffff8c05a9f78700 x1874588654731904/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756479.575034:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588654731904/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756479.575048:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756479.575054:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756479.575061:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654731904:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756479.575073:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654732032:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756479.575120:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756479.575150:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756479/real 1787756479] req@ffff8c05a9f7a680 x1874588654732032/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756479.575185:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588654732032/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756479.575201:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756479.575208:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756479.575214:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654732032:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756479.575226:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654732160:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756479.575266:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756479.575287:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756479/real 1787756479] req@ffff8c05a9f78a80 x1874588654732160/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756479.575308:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588654732160/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756479.575320:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756479.575326:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756479.575332:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654732160:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756479.575344:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654732288:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756479.575380:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756479.575399:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756479/real 1787756479] req@ffff8c05a9f7b480 x1874588654732288/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756479.575417:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588654732288/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756479.575429:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756479.575434:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756479.575438:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654732288:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756479.575448:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654732416:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756479.575503:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756479.575523:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756479/real 1787756479] req@ffff8c05a9f79c00 x1874588654732416/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756479.575552:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79c00 x1874588654732416/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756479.575564:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756479.575569:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756479.575575:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654732416:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756479.575611:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654732544:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756479.575646:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756479.575664:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756479/real 1787756479] req@ffff8c05a9f7aa00 x1874588654732544/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756479.575682:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588654732544/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756479.575694:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756479.575698:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756479.575702:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654732544:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756479.575711:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654732672:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756479.575739:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756479.575755:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756479/real 1787756479] req@ffff8c05a9f79880 x1874588654732672/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756479.575773:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588654732672/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756479.575794:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756479.575799:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756479.575804:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654732672:192.168.201.107@tcp:8:kworker.0 00010000:00010000:2.0:1787756480.150216:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756484.246050:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756485.270128:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756488.342047:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756492.438131:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756493.462140:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787756494.934338:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588654735104/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756494.934365:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756494.934373:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756494.934376:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756494.934384:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9551 00000100:00080000:3.0:1787756494.934390:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756494.934395:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756494.934402:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756494.934423:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588654735232/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756494.934441:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756494.934454:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756494.934460:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756494.934469:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9551 00000100:00080000:3.0:1787756494.934474:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756494.934486:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756494.934498:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756494.934520:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588654735360/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756494.934553:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756494.934558:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756494.934560:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756494.934566:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9551 00000100:00080000:3.0:1787756494.934571:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756494.934576:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756494.934582:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756494.934629:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588654735488/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756494.934646:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756494.934654:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787756494.934662:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756494.934680:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9551 00000100:00080000:3.0:1787756494.934690:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756494.934698:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756494.934713:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756494.934756:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588654735616/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756494.934808:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756494.934826:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756494.934833:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756494.934840:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9551 00000100:00080000:3.0:1787756494.934847:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756494.934853:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756494.934862:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756494.934921:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588654735744/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787756496.534089:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756497.558238:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756499.606136:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756500.054090:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756500.054107:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787756500.054135:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756500.054150:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9556 00000100:00080000:3.0:1787756500.054169:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756500.054180:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787756500.054249:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78000 x1874588654735872/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756500.054295:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756500.054300:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756500.054303:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756500.054308:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9556 00000100:00080000:3.0:1787756500.054314:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756500.054319:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756500.054330:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756500.054349:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588654736000/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756500.054365:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756500.054370:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756500.054372:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756500.054377:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9556 00000100:00080000:3.0:1787756500.054382:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756500.054387:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756500.054394:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756500.054413:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588654736128/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756500.054429:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756500.054434:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756500.054437:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756500.054441:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9556 00000100:00080000:3.0:1787756500.054445:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756500.054450:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756500.054457:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756500.054471:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588654736256/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756500.054482:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756500.054486:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756500.054488:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756500.054492:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9556 00000100:00080000:3.0:1787756500.054500:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756500.054513:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756500.054520:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756500.054537:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588654736384/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756500.054558:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756500.054563:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756500.054570:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756500.054578:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9556 00000100:00080000:3.0:1787756500.054590:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756500.054600:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756500.054611:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756500.054633:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79c00 x1874588654736512/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756500.054651:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756500.054662:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787756500.054670:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756500.054683:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9556 00000100:00080000:3.0:1787756500.054694:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756500.054704:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756500.054718:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756500.054736:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588654736640/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756500.054748:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756500.054755:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756500.054762:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756500.054772:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9556 00000100:00080000:3.0:1787756500.054779:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756500.054791:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756500.054804:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756500.054829:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588654736768/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787756500.630287:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756501.654211:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756504.726284:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756505.173966:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756505.173975:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787756505.173979:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756505.173988:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9561 00000100:00080000:3.0:1787756505.173998:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756505.174005:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787756505.174060:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588654736896/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756505.174088:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756505.174091:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756505.174093:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756505.174097:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9561 00000100:00080000:3.0:1787756505.174101:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756505.174104:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756505.174113:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756505.174126:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588654737024/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756505.174136:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756505.174142:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756505.174144:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756505.174148:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9561 00000100:00080000:3.0:1787756505.174153:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756505.174160:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756505.174170:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756505.174181:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588654737152/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756505.174192:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756505.174196:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756505.174198:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756505.174201:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9561 00000100:00080000:3.0:1787756505.174205:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756505.174208:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756505.174212:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756505.174223:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588654737280/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756505.174241:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756505.174246:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756505.174248:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756505.174254:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9561 00000100:00080000:3.0:1787756505.174258:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756505.174261:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756505.174267:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756505.174279:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79c00 x1874588654737408/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756505.174297:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756505.174310:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756505.174313:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756505.174318:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9561 00000100:00080000:3.0:1787756505.174322:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756505.174329:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756505.174339:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756505.174351:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588654737536/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756505.174370:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756505.174374:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787756505.174378:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756505.174390:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9561 00000100:00080000:3.0:1787756505.174395:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756505.174400:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756505.174407:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756505.174431:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588654737664/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756505.174481:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756505.174493:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756505.174498:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756505.174508:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9561 00000100:00080000:3.0:1787756505.174517:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756505.174530:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756505.174539:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756505.174561:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588654737792/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787756505.750089:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756507.798052:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756508.822218:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756509.846121:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756512.918083:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756513.942050:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756515.990142:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756517.014071:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756518.038059:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756520.534150:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756520.534165:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787756520.534173:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756520.534190:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9576 00000100:00080000:2.0:1787756520.534218:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756520.534233:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756520.534295:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588654739968/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756520.534336:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756520.534353:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756520.534356:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756520.534362:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9576 00000100:00080000:2.0:1787756520.534370:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756520.534376:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756520.534390:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756520.534417:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588654740096/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756520.534431:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756520.534436:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756520.534439:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756520.534444:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9576 00000100:00080000:2.0:1787756520.534449:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756520.534467:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756520.534478:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756520.534508:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588654740224/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756520.534544:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756520.534552:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756520.534555:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756520.534564:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9576 00000100:00080000:2.0:1787756520.534574:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756520.534580:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756520.534590:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756520.534618:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588654740352/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756520.534650:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756520.534658:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756520.534673:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756520.534685:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9576 00000100:00080000:2.0:1787756520.534694:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756520.534700:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756520.534711:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756520.534791:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588654740480/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756520.534820:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756520.534829:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756520.534832:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756520.534843:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9576 00000100:00080000:2.0:1787756520.534860:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756520.534873:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756520.534946:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756520.534979:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588654740608/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756520.535010:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756520.535030:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787756520.535037:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756520.535049:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9576 00000100:00080000:2.0:1787756520.535060:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756520.535073:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756520.535101:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756520.535157:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588654740736/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756520.535185:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756520.535198:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756520.535204:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756520.535231:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9576 00000100:00080000:2.0:1787756520.535245:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756520.535255:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756520.535270:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756520.535304:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588654740864/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787756521.110148:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756522.134098:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756524.182084:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756525.206102:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756525.653951:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756525.653963:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787756525.653967:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756525.653978:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9582 00000100:00080000:2.0:1787756525.653991:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756525.653998:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756525.654045:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588654740992/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756525.654074:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756525.654077:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756525.654078:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756525.654082:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9582 00000100:00080000:2.0:1787756525.654084:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756525.654087:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756525.654097:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756525.654109:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71f80 x1874588654741120/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756525.654116:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756525.654119:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756525.654121:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756525.654124:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9582 00000100:00080000:2.0:1787756525.654126:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756525.654128:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756525.654132:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756525.654141:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588654741248/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756525.654152:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756525.654159:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756525.654160:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756525.654165:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9582 00000100:00080000:2.0:1787756525.654167:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756525.654170:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756525.654175:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756525.654188:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588654741376/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756525.654195:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756525.654198:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756525.654199:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756525.654204:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9582 00000100:00080000:2.0:1787756525.654207:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756525.654209:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756525.654213:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756525.654222:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588654741504/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756525.654233:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756525.654237:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756525.654241:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756525.654246:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9582 00000100:00080000:2.0:1787756525.654252:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756525.654257:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756525.654261:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756525.654269:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588654741632/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756525.654280:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756525.654288:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787756525.654292:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756525.654308:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9582 00000100:00080000:2.0:1787756525.654322:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756525.654334:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756525.654354:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756525.654382:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588654741760/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756525.654394:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756525.654398:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756525.654400:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756525.654414:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9582 00000100:00080000:2.0:1787756525.654421:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756525.654426:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756525.654436:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756525.654454:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588654741888/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787756526.230281:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756529.302264:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756530.326107:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756530.774029:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756530.774041:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787756530.774048:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756530.774060:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9587 00000100:00080000:2.0:1787756530.774073:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756530.774083:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756530.774126:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588654742016/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756530.774170:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756530.774175:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756530.774177:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756530.774183:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9587 00000100:00080000:2.0:1787756530.774187:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756530.774192:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756530.774203:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756530.774219:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73800 x1874588654742144/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756530.774234:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756530.774238:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756530.774241:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756530.774245:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9587 00000100:00080000:2.0:1787756530.774250:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756530.774259:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787756530.774262:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654742016:192.168.201.107@tcp:250:kworker.0 00000008:00100000:2.0:1787756530.774266:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756530.774290:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588654742272/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756530.774314:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756530.774319:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756530.774321:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756530.774327:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9587 00000100:00080000:2.0:1787756530.774349:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756530.774354:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756530.774360:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787756530.774371:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756530.774380:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588654742400/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756530.774396:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756530.774401:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756530.774403:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756530.774409:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9587 00000100:00080000:2.0:1787756530.774415:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756530.774421:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756530.774428:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756530.774429:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756530/real 1787756530] req@ffff8c05b2e70380 x1874588654742016/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756530.774451:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588654742528/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787756530.774466:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588654742016/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756530.774468:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756530.774484:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756530.774492:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756530.774508:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756530.774517:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787756530.774519:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9587 00000100:00100000:3.0:1787756530.774527:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654742016:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787756530.774530:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756530.774545:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787756530.774552:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654742144:192.168.201.107@tcp:38:kworker.0 00000008:00100000:2.0:1787756530.774555:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756530.774608:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588654742656/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756530.774625:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756530.774630:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787756530.774633:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787756530.774639:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787756530.774647:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9587 00000100:00080000:2.0:1787756530.774657:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756530.774664:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787756530.774672:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756530/real 1787756530] req@ffff8c05b2e73800 x1874588654742144/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787756530.774677:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756530.774708:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588654742784/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787756530.774709:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73800 x1874588654742144/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756530.774770:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756530.774771:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756530.774787:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756530.774788:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787756530.774797:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787756530.774802:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654742144:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787756530.774811:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9587 00000100:00080000:2.0:1787756530.774820:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756530.774825:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756530.774832:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756530.774846:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588654742912/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787756530.774850:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654742272:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787756530.774962:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756530.774992:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756530/real 1787756530] req@ffff8c05b2e71880 x1874588654742272/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756530.775020:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588654742272/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756530.775032:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756530.775039:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756530.775043:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654742272:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756530.775054:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654742400:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787756530.775086:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756530.775101:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756530/real 1787756530] req@ffff8c05b2e71c00 x1874588654742400/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756530.775125:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588654742400/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756530.775135:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756530.775138:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756530.775144:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654742400:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787756530.775151:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654742528:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787756530.775170:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756530.775181:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756530/real 1787756530] req@ffff8c05b2e73b80 x1874588654742528/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756530.775197:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588654742528/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756530.775206:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756530.775209:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756530.775213:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654742528:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756530.775222:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654742656:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787756530.775242:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756530.775253:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756530/real 1787756530] req@ffff8c05b2e71500 x1874588654742656/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756530.775267:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588654742656/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756530.775284:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756530.775288:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756530.775292:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654742656:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756530.775298:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654742784:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787756530.775315:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756530.775326:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756530/real 1787756530] req@ffff8c05b2e73480 x1874588654742784/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756530.775339:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588654742784/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756530.775348:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756530.775351:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756530.775355:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654742784:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787756530.775376:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654742912:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787756530.775395:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756530.775405:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756530/real 1787756530] req@ffff8c05b2e70700 x1874588654742912/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756530.775419:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588654742912/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756530.775428:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756530.775432:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756530.775436:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654742912:192.168.201.107@tcp:8:kworker.0 00010000:00010000:3.0:1787756532.374108:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756533.398069:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756534.422155:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756535.894018:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756535.894029:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787756535.894033:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756535.894043:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9592 00000100:00080000:3.0:1787756535.894053:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756535.894060:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787756535.894094:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588654743040/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756535.894126:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756535.894129:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756535.894131:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756535.894136:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9592 00000100:00080000:3.0:1787756535.894139:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756535.894143:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756535.894151:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756535.894164:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78000 x1874588654743168/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756535.894173:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756535.894177:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756535.894178:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756535.894182:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9592 00000100:00080000:3.0:1787756535.894185:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756535.894189:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756535.894194:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756535.894202:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588654743296/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756535.894212:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756535.894217:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756535.894220:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756535.894227:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9592 00000100:00080000:3.0:1787756535.894235:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756535.894240:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756535.894246:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756535.894257:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588654743424/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756535.894269:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756535.894274:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756535.894278:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756535.894291:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9592 00000100:00080000:3.0:1787756535.894304:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756535.894315:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756535.894320:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756535.894337:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588654743552/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756535.894347:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756535.894352:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756535.894354:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756535.894359:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9592 00000100:00080000:3.0:1787756535.894365:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756535.894371:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756535.894378:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756535.894393:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588654743680/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756535.894403:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756535.894407:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787756535.894409:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756535.894414:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9592 00000100:00080000:3.0:1787756535.894419:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756535.894423:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756535.894429:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756535.894447:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588654743808/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756535.894461:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756535.894465:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756535.894467:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756535.894473:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9592 00000100:00080000:3.0:1787756535.894478:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756535.894494:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756535.894499:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756535.894510:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588654743936/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787756537.494143:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756538.518006:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756540.566157:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756541.013998:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756541.014011:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787756541.014017:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756541.014030:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9597 00000100:00080000:3.0:1787756541.014043:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756541.014053:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787756541.014100:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588654744064/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756541.014140:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756541.014144:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756541.014147:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756541.014152:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9597 00000100:00080000:3.0:1787756541.014156:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756541.014160:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756541.014172:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756541.014194:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588654744192/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756541.014207:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756541.014212:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756541.014216:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756541.014221:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9597 00000100:00080000:3.0:1787756541.014225:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756541.014232:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756541.014248:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756541.014281:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588654744320/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756541.014300:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756541.014305:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756541.014310:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756541.014316:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9597 00000100:00080000:3.0:1787756541.014323:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756541.014328:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756541.014336:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756541.014371:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588654744448/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756541.014383:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756541.014388:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756541.014419:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756541.014427:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9597 00000100:00080000:3.0:1787756541.014433:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756541.014440:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756541.014448:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756541.014472:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588654744576/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756541.014487:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756541.014497:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756541.014510:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756541.014523:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9597 00000100:00080000:3.0:1787756541.014530:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756541.014537:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756541.014546:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756541.014566:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588654744704/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756541.014586:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756541.014602:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787756541.014606:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756541.014616:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9597 00000100:00080000:3.0:1787756541.014627:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756541.014633:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756541.014642:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756541.014662:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588654744832/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756541.014676:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756541.014683:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756541.014686:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756541.014707:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9597 00000100:00080000:3.0:1787756541.014715:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756541.014723:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756541.014732:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756541.014766:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588654744960/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787756541.590178:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756542.614117:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787756544.086327:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.107:988 00000400:00000100:2.0:1787756544.086447:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.107@tcp at host 192.168.201.107 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787756544.086480:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787756544.086565:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787756544.086586:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787756544.086606:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787756544.086625:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787756544.086646:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787756544.086669:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787756544.086694:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00010000:00010000:2.0:1787756545.686164:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756546.134003:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756546.134017:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787756546.134022:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756546.134036:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9602 00000100:00080000:3.0:1787756546.134049:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756546.134058:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787756546.134106:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78000 x1874588654745088/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756546.134144:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756546.134148:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756546.134151:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756546.134156:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9602 00000100:00080000:3.0:1787756546.134160:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756546.134164:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756546.134174:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756546.134191:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588654745216/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756546.134206:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756546.134211:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756546.134214:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756546.134218:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9602 00000100:00080000:3.0:1787756546.134222:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756546.134227:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756546.134233:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756546.134249:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588654745344/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756546.134271:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756546.134275:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756546.134277:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756546.134281:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9602 00000100:00080000:3.0:1787756546.134285:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756546.134289:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756546.134295:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756546.134347:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588654745472/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756546.134368:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756546.134375:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756546.134378:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756546.134385:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9602 00000100:00080000:3.0:1787756546.134394:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756546.134406:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756546.134423:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756546.134453:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588654745600/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756546.134475:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756546.134482:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756546.134491:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756546.134508:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9602 00000100:00080000:3.0:1787756546.134524:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756546.134537:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756546.134548:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756546.134578:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79c00 x1874588654745728/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756546.134616:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756546.134627:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787756546.134636:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756546.134653:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9602 00000100:00080000:3.0:1787756546.134665:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756546.134678:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756546.134689:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756546.134715:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588654745856/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756546.134750:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756546.134756:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756546.134761:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756546.134776:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9602 00000100:00080000:3.0:1787756546.134791:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756546.134807:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756546.134821:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756546.134857:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588654745984/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787756546.710136:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756548.758105:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756549.782125:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756550.806143:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756551.254152:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756551.254163:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787756551.254168:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756551.254178:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9607 00000100:00080000:2.0:1787756551.254190:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756551.254198:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756551.254278:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71f80 x1874588654746112/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756551.254335:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756551.254341:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756551.254345:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756551.254351:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9607 00000100:00080000:2.0:1787756551.254356:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756551.254360:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756551.254371:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756551.254387:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588654746240/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756551.254399:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756551.254403:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756551.254405:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756551.254409:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9607 00000100:00080000:2.0:1787756551.254414:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756551.254423:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756551.254435:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756551.254446:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588654746368/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756551.254457:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756551.254461:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756551.254463:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756551.254468:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9607 00000100:00080000:2.0:1787756551.254475:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756551.254480:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756551.254484:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756551.254495:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588654746496/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756551.254507:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756551.254516:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756551.254519:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756551.254526:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9607 00000100:00080000:2.0:1787756551.254530:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756551.254535:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756551.254540:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756551.254552:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588654746624/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756551.254564:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756551.254575:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756551.254581:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756551.254594:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9607 00000100:00080000:2.0:1787756551.254599:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756551.254606:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756551.254613:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756551.254632:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588654746752/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756551.254684:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756551.254691:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787756551.254694:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756551.254699:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9607 00000100:00080000:2.0:1787756551.254705:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756551.254710:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756551.254715:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756551.254733:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588654746880/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756551.254755:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756551.254765:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756551.254768:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756551.254776:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9607 00000100:00080000:2.0:1787756551.254785:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756551.254795:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756551.254826:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756551.254847:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588654747008/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787756553.878117:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756554.902184:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756556.373957:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756556.373965:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787756556.373968:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756556.373976:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9612 00000100:00080000:2.0:1787756556.373985:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756556.373991:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756556.374022:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588654747136/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756556.374044:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756556.374048:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756556.374049:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756556.374053:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9612 00000100:00080000:2.0:1787756556.374057:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756556.374060:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756556.374067:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756556.374079:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654747136:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787756556.374082:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588654747264/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756556.374094:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756556.374097:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756556.374098:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756556.374102:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9612 00000100:00080000:2.0:1787756556.374105:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756556.374107:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756556.374111:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756556.374136:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588654747392/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756556.374145:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756556.374149:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756556.374151:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756556.374155:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9612 00000800:00000100:3.0:1787756556.374159:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787756556.374166:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756556.374171:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756556.374177:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756556.374188:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588654747520/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756556.374195:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756556.374200:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1787756556.374200:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756556/real 1787756556] req@ffff8c05b2e70a80 x1874588654747136/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756556.374204:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756556.374212:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9612 00000100:00080000:2.0:1787756556.374215:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756556.374218:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787756556.374223:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588654747136/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787756556.374224:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787756556.374238:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756556.374242:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756556.374245:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588654747648/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787756556.374247:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654747136:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787756556.374263:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756556.374267:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756556.374270:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756556.374275:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9612 00000100:00080000:2.0:1787756556.374279:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756556.374282:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756556.374287:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756556.374296:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588654747776/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756556.374302:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756556.374305:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787756556.374306:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756556.374309:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9612 00000100:00080000:2.0:1787756556.374312:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756556.374314:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756556.374317:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756556.374324:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654747264:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756556.374325:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588654747904/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756556.374337:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756556.374340:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756556.374341:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756556.374344:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9612 00000100:00080000:2.0:1787756556.374347:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787756556.374351:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787756556.374352:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756556.374358:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756556.374367:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756556/real 1787756556] req@ffff8c05b2e72300 x1874588654747264/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756556.374369:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588654748032/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787756556.374407:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588654747264/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756556.374414:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756556.374417:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756556.374420:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654747264:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756556.374425:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654747392:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787756556.374437:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756556.374443:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756556/real 1787756556] req@ffff8c05b2e71180 x1874588654747392/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756556.374451:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588654747392/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756556.374456:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756556.374458:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756556.374460:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654747392:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756556.374464:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654747520:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787756556.374474:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756556.374572:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756556/real 1787756556] req@ffff8c05b2e72a00 x1874588654747520/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756556.374582:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588654747520/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756556.374593:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756556.374596:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756556.374599:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654747520:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787756556.374605:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654747648:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787756556.374659:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756556.374668:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756556/real 1787756556] req@ffff8c05b2e70e00 x1874588654747648/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756556.374676:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588654747648/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756556.374681:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756556.374683:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756556.374685:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654747648:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756556.374689:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654747776:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787756556.374701:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756556.374707:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756556/real 1787756556] req@ffff8c05b2e72680 x1874588654747776/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756556.374714:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588654747776/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756556.374718:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756556.374720:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756556.374722:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654747776:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756556.374735:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654747904:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787756556.374745:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756556.374751:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756556/real 1787756556] req@ffff8c05b2e72d80 x1874588654747904/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756556.374758:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588654747904/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756556.374763:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756556.374765:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756556.374767:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654747904:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787756556.374770:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654748032:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787756556.374779:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756556.374793:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756556/real 1787756556] req@ffff8c05b2e73100 x1874588654748032/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756556.374801:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588654748032/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756556.374806:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756556.374807:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756556.374809:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654748032:192.168.201.107@tcp:8:kworker.0 00010000:00010000:3.0:1787756556.950084:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756557.974199:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756558.998053:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756561.494013:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756561.494025:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787756561.494056:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756561.494081:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9617 00000100:00080000:2.0:1787756561.494094:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756561.494103:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756561.494146:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588654748160/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756561.494182:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756561.494189:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756561.494193:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756561.494201:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9617 00000100:00080000:2.0:1787756561.494207:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756561.494215:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756561.494230:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756561.494256:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71f80 x1874588654748288/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787756561.494259:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654748160:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787756561.494276:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756561.494282:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756561.494286:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756561.494293:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9617 00000100:00080000:2.0:1787756561.494298:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756561.494306:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756561.494314:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756561.494332:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588654748416/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787756561.494348:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787756561.494350:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756561.494356:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756561.494359:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756561.494365:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9617 00000100:00080000:2.0:1787756561.494370:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756561.494377:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756561.494386:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756561.494406:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588654748544/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787756561.494406:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756561/real 1787756561] req@ffff8c05b2e70000 x1874588654748160/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756561.494426:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787756561.494437:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588654748160/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756561.494451:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756561.494454:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756561.494456:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756561.494461:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9617 00000100:00080000:3.0:1787756561.494464:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787756561.494467:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787756561.494471:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654748160:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787756561.494474:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756561.494483:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756561.494502:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588654748672/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787756561.494507:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654748288:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787756561.494523:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756561.494529:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756561.494532:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756561.494540:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9617 00000100:00080000:2.0:1787756561.494546:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787756561.494548:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787756561.494553:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756561.494562:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756561.494571:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756561/real 1787756561] req@ffff8c05b2e71f80 x1874588654748288/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756561.494581:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588654748800/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787756561.494597:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71f80 x1874588654748288/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756561.494601:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756561.494608:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787756561.494611:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756561.494614:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756561.494619:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9617 00000100:00080000:3.0:1787756561.494620:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787756561.494625:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787756561.494627:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654748288:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787756561.494632:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787756561.494639:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654748416:192.168.201.107@tcp:38:kworker.0 00000008:00100000:2.0:1787756561.494640:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756561.494675:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588654748928/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787756561.494675:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787756561.494693:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787756561.494695:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756561/real 1787756561] req@ffff8c05b2e70700 x1874588654748416/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756561.494712:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756561.494715:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756561.494723:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9617 00000100:00080000:2.0:1787756561.494729:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756561.494736:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756561.494745:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756561.494749:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588654748416/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756561.494764:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588654749056/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756561.494766:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756561.494772:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756561.494779:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654748416:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756561.494791:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654748544:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787756561.494831:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756561.494853:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756561/real 1787756561] req@ffff8c05b2e73480 x1874588654748544/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756561.494869:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588654748544/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756561.494934:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756561.494943:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756561.494951:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654748544:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787756561.494982:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654748672:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787756561.495014:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756561.495057:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756561/real 1787756561] req@ffff8c05b2e71500 x1874588654748672/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756561.495077:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588654748672/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756561.495096:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756561.495100:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756561.495104:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654748672:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756561.495112:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654748800:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787756561.495140:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756561.495154:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756561/real 1787756561] req@ffff8c05b2e73b80 x1874588654748800/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756561.495168:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588654748800/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756561.495177:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756561.495181:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756561.495185:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654748800:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756561.495191:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654748928:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787756561.495210:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756561.495220:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756561/real 1787756561] req@ffff8c05b2e71c00 x1874588654748928/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756561.495234:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588654748928/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756561.495243:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756561.495247:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756561.495251:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654748928:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787756561.495257:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654749056:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787756561.495275:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756561.495286:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756561/real 1787756561] req@ffff8c05b2e71880 x1874588654749056/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756561.495298:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588654749056/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756561.495307:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756561.495310:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756561.495314:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654749056:192.168.201.107@tcp:8:kworker.0 00010000:00010000:2.0:1787756562.070210:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756563.094132:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756565.142162:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756566.166157:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756566.613971:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756566.613979:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787756566.613983:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756566.613990:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9623 00000100:00080000:3.0:1787756566.613999:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756566.614005:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787756566.614036:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588654749184/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756566.614058:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756566.614060:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756566.614062:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756566.614065:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9623 00000100:00080000:3.0:1787756566.614067:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756566.614071:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756566.614078:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756566.614088:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588654749312/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756566.614097:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756566.614105:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756566.614107:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756566.614111:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9623 00000100:00080000:3.0:1787756566.614113:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756566.614115:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756566.614119:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756566.614127:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588654749440/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756566.614135:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756566.614138:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756566.614139:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756566.614142:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9623 00000100:00080000:3.0:1787756566.614144:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756566.614146:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756566.614149:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756566.614156:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588654749568/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756566.614170:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756566.614173:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756566.614175:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756566.614179:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9623 00000100:00080000:3.0:1787756566.614182:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756566.614185:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756566.614188:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756566.614195:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79c00 x1874588654749696/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756566.614202:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756566.614206:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756566.614209:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756566.614216:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9623 00000100:00080000:3.0:1787756566.614221:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756566.614224:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756566.614228:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756566.614236:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588654749824/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756566.614252:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756566.614260:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787756566.614264:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756566.614274:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9623 00000100:00080000:3.0:1787756566.614279:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756566.614284:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756566.614289:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756566.614303:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588654749952/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756566.614312:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756566.614315:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756566.614316:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756566.614324:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9623 00000100:00080000:3.0:1787756566.614328:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756566.614331:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756566.614335:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756566.614348:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588654750080/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787756567.190178:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756570.262153:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756571.286153:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756571.734037:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756571.734050:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787756571.734055:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756571.734068:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9628 00000100:00080000:2.0:1787756571.734080:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756571.734090:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756571.734136:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73800 x1874588654750208/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756571.734173:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756571.734178:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756571.734180:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756571.734185:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9628 00000100:00080000:2.0:1787756571.734189:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756571.734194:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756571.734204:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756571.734220:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588654750336/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756571.734234:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756571.734239:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756571.734244:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756571.734249:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9628 00000100:00080000:2.0:1787756571.734253:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756571.734263:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756571.734277:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756571.734296:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588654750464/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756571.734311:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756571.734316:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756571.734319:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756571.734325:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9628 00000100:00080000:2.0:1787756571.734330:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756571.734334:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756571.734340:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756571.734355:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588654750592/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756571.734377:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756571.734383:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756571.734386:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756571.734394:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9628 00000100:00080000:2.0:1787756571.734399:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756571.734418:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756571.734426:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756571.734441:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588654750720/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756571.734455:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756571.734459:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756571.734462:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756571.734471:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9628 00000100:00080000:2.0:1787756571.734481:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756571.734494:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756571.734508:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756571.734525:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588654750848/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756571.734565:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756571.734572:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787756571.734578:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756571.734593:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9628 00000100:00080000:2.0:1787756571.734602:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756571.734616:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756571.734624:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756571.734647:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588654750976/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756571.734663:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756571.734668:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756571.734670:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756571.734681:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9628 00000100:00080000:2.0:1787756571.734692:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756571.734698:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756571.734706:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756571.734728:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588654751104/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787756573.334090:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756574.358182:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756575.382226:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756576.854157:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756576.854174:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787756576.854182:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756576.854216:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9633 00000100:00080000:3.0:1787756576.854232:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756576.854244:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787756576.854294:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588654751232/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756576.854342:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756576.854347:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756576.854350:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756576.854355:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9633 00000100:00080000:3.0:1787756576.854360:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756576.854365:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756576.854376:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756576.854400:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78000 x1874588654751360/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756576.854420:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756576.854427:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756576.854430:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756576.854437:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9633 00000100:00080000:3.0:1787756576.854444:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756576.854457:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756576.854476:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756576.854506:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588654751488/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756576.854530:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756576.854537:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756576.854541:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756576.854549:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9633 00000100:00080000:3.0:1787756576.854559:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756576.854570:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756576.854586:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756576.854615:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588654751616/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756576.854635:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756576.854641:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756576.854645:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756576.854656:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9633 00000100:00080000:3.0:1787756576.854665:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756576.854681:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756576.854698:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756576.854746:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588654751744/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756576.854810:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756576.854820:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756576.854826:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756576.854845:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9633 00000100:00080000:3.0:1787756576.854855:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756576.854863:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756576.854932:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756576.854960:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588654751872/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756576.855091:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756576.855104:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787756576.855108:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756576.855119:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9633 00000100:00080000:3.0:1787756576.855129:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756576.855138:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756576.855149:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756576.855181:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588654752000/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756576.855202:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756576.855213:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756576.855221:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756576.855230:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9633 00000100:00080000:3.0:1787756576.855239:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756576.855250:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756576.855258:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756576.855284:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588654752128/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787756578.454114:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756579.478109:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756581.526021:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756581.973959:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756581.973967:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787756581.973971:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756581.973980:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9638 00000100:00080000:3.0:1787756581.973988:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756581.973993:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787756581.974037:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588654752256/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756581.974060:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756581.974063:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756581.974064:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756581.974067:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9638 00000100:00080000:3.0:1787756581.974069:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756581.974072:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756581.974079:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756581.974090:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588654752384/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756581.974097:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756581.974102:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756581.974105:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756581.974114:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9638 00000100:00080000:3.0:1787756581.974116:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756581.974118:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756581.974121:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756581.974129:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588654752512/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756581.974140:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756581.974142:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756581.974144:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756581.974156:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9638 00000100:00080000:3.0:1787756581.974158:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756581.974161:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756581.974163:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756581.974169:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588654752640/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756581.974175:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756581.974178:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756581.974179:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756581.974181:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9638 00000100:00080000:3.0:1787756581.974183:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756581.974190:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756581.974195:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756581.974202:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588654752768/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756581.974209:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756581.974211:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756581.974212:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756581.974214:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9638 00000100:00080000:3.0:1787756581.974217:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756581.974220:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756581.974223:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756581.974231:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588654752896/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756581.974239:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756581.974246:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787756581.974247:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756581.974255:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9638 00000100:00080000:3.0:1787756581.974259:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756581.974265:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756581.974268:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756581.974275:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588654753024/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756581.974283:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756581.974289:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756581.974291:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756581.974299:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9638 00000100:00080000:3.0:1787756581.974305:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756581.974309:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756581.974312:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756581.974321:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588654753152/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787756582.550103:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756583.574146:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756586.646150:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756587.670305:0:225883:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 101/101 locks to regular blp list, 1 blwis in pool 00010000:00010000:3.0:1787756587.670354:0:34913:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 101 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1787756587.670373:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2e9dc00/0x109d23f2ff214bf8 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x83:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b32536578a8c6 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787756587.670389:0:225883:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 28/28 locks to regular blp list, 1 blwis in pool 00000080:00200000:3.0:1787756587.670426:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x83:0x0]: losing layout lock 00010000:00010000:3.0:1787756587.670525:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2e9c400/0x109d23f2ff214bea lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b32536578a8a3 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787756587.670535:0:225883:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 26/26 locks to regular blp list, 1 blwis in pool 00000080:00200000:3.0:1787756587.670596:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x4:0x0]: losing layout lock 00010000:00010000:3.0:1787756587.670705:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2e9c200/0x109d23f2ff214bdc lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b32536578a879 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787756587.670721:0:225883:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 25/25 locks to regular blp list, 1 blwis in pool 00010000:00010000:3.0:1787756587.670753:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0591285000/0x109d23f2ff214bc7 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x15:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b32536578a84f expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787756587.670758:0:225883:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/26 locks to regular blp list, 2 blwis in pool 00010000:00010000:3.0:1787756587.670783:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b3363600/0x109d23f2ff214bb9 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x5e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b32536578a82c expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787756587.670786:0:15681:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 25 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:2.0:1787756587.670806:0:15681:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8c0583595800 lock: 000000009f286827/0x109d23f2ff214bf1 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9b2b32536578a8aa expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787756587.670825:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x5e:0x0]: losing layout lock 00010000:00010000:3.0:1787756587.670879:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbba9e00/0x109d23f2ff214bab lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x65:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b32536578a809 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787756587.670920:0:15681:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x4:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787756587.670933:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x65:0x0]: losing layout lock 00000080:00200000:2.0:1787756587.670950:0:15681:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x4:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787756587.670979:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2d7be00/0x109d23f2ff214b9d lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b32536578a7e6 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787756587.670998:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x1:0x0]: losing layout lock 00010000:00010000:3.0:1787756587.671026:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2d7b600/0x109d23f2ff214b8f lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x64:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b32536578a7c3 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787756587.671049:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x64:0x0]: losing layout lock 00010000:00010000:3.0:1787756587.671067:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2e9da00/0x109d23f2ff214b81 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x80:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b32536578a7a0 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787756587.671083:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x80:0x0]: losing layout lock 00010000:00010000:3.0:1787756587.671103:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2e9d600/0x109d23f2ff214b7a lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x162:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b32536578a784 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787756587.671114:0:15681:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8c0583595800 lock: 0000000059bd96ce/0x109d23f2ff214bc0 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x27:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9b2b32536578a833 expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787756587.671149:0:15681:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x5e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:3.0:1787756587.671149:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0592633800/0x109d23f2ff214b6c lrc: 2/0,0 mode: PR/PR res: [0x200000bd0:0x6b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b32536578a768 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787756587.671162:0:15681:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x5e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1787756587.671187:0:15681:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8c0583595800 lock: 000000006fe73ec1/0x109d23f2ff214ba4 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9b2b32536578a7ed expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.671194:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0592633a00/0x109d23f2ff214b65 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x162:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b32536578a75a expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787756587.671210:0:15681:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1787756587.671225:0:15681:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1787756587.671259:0:15681:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8c0583595800 lock: 0000000080d18de9/0x109d23f2ff214b96 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x2a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9b2b32536578a7ca expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.671266:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0592633400/0x109d23f2ff214b5e lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x22:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b32536578a73e expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787756587.671272:0:15681:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x64:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1787756587.671284:0:15681:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x64:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1787756587.671308:0:15681:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8c0583595800 lock: 00000000311f3571/0x109d23f2ff214abd lrc: 2/0,0 mode: PR/PR res: [0x280000401:0xf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9b2b32536578a5af expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.671312:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0592632200/0x109d23f2ff214b50 lrc: 2/0,0 mode: PR/PR res: [0x200000400:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b32536578a722 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787756587.671322:0:15681:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x2d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1787756587.671346:0:15681:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x2d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1787756587.671368:0:15681:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8c0583595800 lock: 000000000a48c252/0x109d23f2ff214aa8 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9b2b32536578a585 expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.671378:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0592632e00/0x109d23f2ff214b49 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x22:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b32536578a714 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787756587.671381:0:15681:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x13:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1787756587.671394:0:15681:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x13:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1787756587.671414:0:15681:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8c0583595800 lock: 00000000c8e7e804/0x109d23f2ff214a9a lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x28:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9b2b32536578a562 expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787756587.671439:0:15681:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x60:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1787756587.671451:0:15681:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x60:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787756587.671454:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0592632000/0x109d23f2ff214b3b lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x59:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b32536578a6f1 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787756587.671486:0:15681:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8c0583595800 lock: 00000000a7a8a078/0x109d23f2ff214a8c lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x1e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9b2b32536578a53f expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787756587.671510:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x59:0x0]: losing layout lock 00000080:00200000:2.0:1787756587.671530:0:15681:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x4d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1787756587.671538:0:15681:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x4d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787756587.671543:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0592633000/0x109d23f2ff214b2d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x57:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b32536578a6ce expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787756587.671562:0:15681:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8c0583595800 lock: 00000000eb892072/0x109d23f2ff214a70 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x11:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9b2b32536578a4f9 expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787756587.671623:0:15681:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x31:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787756587.671625:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x57:0x0]: losing layout lock 00000080:00200000:2.0:1787756587.671639:0:15681:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x31:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787756587.671658:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0592633200/0x109d23f2ff214b26 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x7c:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b32536578a6b2 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787756587.671667:0:15681:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8c0583595800 lock: 00000000a1031292/0x109d23f2ff214a3f lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x29:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9b2b32536578a482 expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787756587.671690:0:15681:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x62:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:3.0:1787756587.671690:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0592632600/0x109d23f2ff214b18 lrc: 2/0,0 mode: PR/PR res: [0x200000400:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b32536578a696 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787756587.671701:0:15681:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x62:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1787756587.671724:0:15681:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8c0583595800 lock: 000000009ab4e9fe/0x109d23f2ff2149f2 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x472:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9b2b32536578a3c5 expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.671730:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0592632400/0x109d23f2ff214b11 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x7c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b32536578a688 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787756587.671763:0:15681:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x164:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1787756587.671775:0:15681:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x164:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787756587.671790:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0592633600/0x109d23f2ff214b0a lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x20:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b32536578a66c expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787756587.671794:0:15681:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8c0583595800 lock: 00000000993c1773/0x109d23f2ff2149c8 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x25:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9b2b32536578a35c expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.671821:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0589f74a00/0x109d23f2ff214afc lrc: 2/0,0 mode: PR/PR res: [0x200000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b32536578a650 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787756587.671829:0:15681:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x5a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1787756587.671841:0:15681:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x5a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1787756587.671861:0:15681:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8c0583595800 lock: 000000000a6856e8/0x109d23f2ff2149ba lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x34:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9b2b32536578a339 expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.671900:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0589f75000/0x109d23f2ff214af5 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x20:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b32536578a642 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787756587.671934:0:15681:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x77:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1787756587.671945:0:15681:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x77:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787756587.671961:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0589f74e00/0x109d23f2ff214ae7 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x168:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b32536578a61f expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787756587.671970:0:15681:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8c0583595800 lock: 00000000de107905/0x109d23f2ff214974 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x35:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9b2b32536578a28a expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787756587.671978:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x168:0x0]: losing layout lock 00010000:00010000:3.0:1787756587.672001:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0589f75a00/0x109d23f2ff214ad9 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b32536578a5f5 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787756587.672008:0:15681:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x7b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1787756587.672021:0:15681:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x7b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787756587.672045:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0589f74400/0x109d23f2ff214ac4 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x16:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b32536578a5cb expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787756587.672046:0:15681:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8c0583595800 lock: 0000000001e3b349/0x109d23f2ff21495f lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9b2b32536578a24b expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787756587.672085:0:15681:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1787756587.672093:0:15681:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787756587.672094:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0589f75c00/0x109d23f2ff214ab6 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b32536578a5a8 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787756587.672105:0:15681:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8c0583595800 lock: 0000000023450f2b/0x109d23f2ff214935 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x12:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9b2b32536578a1e2 expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787756587.672108:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x2d:0x0]: losing layout lock 00010000:00010000:3.0:1787756587.672125:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0589f74800/0x109d23f2ff214aa1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b32536578a57e expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787756587.672147:0:15681:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x34:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787756587.672154:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x13:0x0]: losing layout lock 00000080:00200000:2.0:1787756587.672160:0:15681:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x34:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787756587.672171:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2d7a200/0x109d23f2ff214a93 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x60:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b32536578a55b expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787756587.672180:0:15681:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8c0583595800 lock: 00000000b4388321/0x109d23f2ff214927 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x2b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9b2b32536578a1bf expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787756587.672200:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x60:0x0]: losing layout lock 00000080:00200000:2.0:1787756587.672216:0:15681:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x66:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:3.0:1787756587.672228:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbace400/0x109d23f2ff214a85 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b32536578a538 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787756587.672229:0:15681:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x66:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1787756587.672251:0:15681:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8c0583595800 lock: 0000000059ed1420/0x109d23f2ff214919 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x24:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9b2b32536578a19c expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787756587.672269:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x4d:0x0]: losing layout lock 00000080:00200000:2.0:1787756587.672277:0:15681:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x58:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:3.0:1787756587.672293:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbacf400/0x109d23f2ff214a77 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x26:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b32536578a515 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787756587.672295:0:15681:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x58:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1787756587.672322:0:15681:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8c0583595800 lock: 00000000b289b1c6/0x109d23f2ff2148fd lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x2c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9b2b32536578a156 expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787756587.672336:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x26:0x0]: losing layout lock 00010000:00010000:3.0:1787756587.672358:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbacf000/0x109d23f2ff214a62 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b32536578a4eb expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787756587.672364:0:15681:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x67:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1787756587.672378:0:15681:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x67:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:3.0:1787756587.672399:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x31:0x0]: losing layout lock 00010000:00010000:2.0:1787756587.672404:0:15681:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8c0583595800 lock: 0000000099e050e6/0x109d23f2ff2148e1 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x39:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9b2b32536578a110 expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.672424:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbacea00/0x109d23f2ff214a4d lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1b79:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b32536578a4ac expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787756587.672442:0:15681:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x2a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1787756587.672456:0:15681:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x2a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1787756587.672481:0:15681:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8c0583595800 lock: 00000000eeee5529/0x109d23f2ff214886 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9b2b32536578a030 expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.672487:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbacf800/0x109d23f2ff214a46 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1b79:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b32536578a49e expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787756587.672497:0:15681:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1787756587.672509:0:15681:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1787756587.672528:0:15681:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8c0583595800 lock: 000000001efae11a/0x109d23f2ff21480f lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x473:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9b2b325365789f03 expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.672551:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbacfa00/0x109d23f2ff214a38 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x62:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b32536578a47b expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787756587.672606:0:15681:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x167:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787756587.672620:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x62:0x0]: losing layout lock 00000080:00200000:2.0:1787756587.672640:0:15681:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x167:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787756587.672662:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbacfc00/0x109d23f2ff214a2a lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b32536578a458 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787756587.672668:0:15681:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8c0583595800 lock: 000000002e528c15/0x109d23f2ff21479f lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x15:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9b2b325365789de4 expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787756587.672700:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x5d:0x0]: losing layout lock 00000080:00200000:2.0:1787756587.672708:0:15681:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x39:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1787756587.672723:0:15681:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x39:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787756587.672742:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbacec00/0x109d23f2ff214a1c lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b32536578a42e expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787756587.672750:0:15681:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8c0583595800 lock: 000000001bf711a1/0x109d23f2ff214767 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x26:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9b2b325365789d51 expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.672785:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbad7c00/0x109d23f2ff214a07 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x1e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b32536578a404 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787756587.672793:0:15681:0:(vvp_io.c:1826:vvp_io_init()) [0x200000402:0x5c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1787756587.672810:0:15681:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x5c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787756587.672829:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbad6200/0x109d23f2ff2149f9 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x79:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b32536578a3e1 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787756587.672835:0:15681:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8c0583595800 lock: 00000000363d3556/0x109d23f2ff214759 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9b2b325365789d2e expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787756587.672879:0:15681:0:(vvp_io.c:1826:vvp_io_init()) [0x200000403:0x2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1787756587.672892:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x79:0x0]: losing layout lock 00010000:00010000:3.0:1787756587.672918:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbad7a00/0x109d23f2ff2149eb lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x164:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b32536578a3be expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787756587.672929:0:15681:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00000080:00200000:3.0:1787756587.672934:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x164:0x0]: losing layout lock 00010000:00010000:3.0:1787756587.672963:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbad7600/0x109d23f2ff2149dd lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x76:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b32536578a39b expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787756587.672974:0:15681:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8c0583595800 lock: 000000009f286827/0x109d23f2ff214bf1 lrc: 1/0,0 mode: --/PR res: [0x280000401:0x3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b32536578a8aa expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787756587.672989:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x76:0x0]: losing layout lock 00010000:00010000:2.0:1787756587.673010:0:15681:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8c0583595800 lock: 0000000059bd96ce/0x109d23f2ff214bc0 lrc: 1/0,0 mode: --/PR res: [0x280000401:0x27:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b32536578a833 expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.673023:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbad6a00/0x109d23f2ff2149cf lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x5b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b32536578a378 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787756587.673025:0:15681:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8c0583595800 lock: 000000006fe73ec1/0x109d23f2ff214ba4 lrc: 1/0,0 mode: --/PR res: [0x280000401:0x4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b32536578a7ed expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787756587.673062:0:15681:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8c0583595800 lock: 0000000080d18de9/0x109d23f2ff214b96 lrc: 1/0,0 mode: --/PR res: [0x280000401:0x2a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b32536578a7ca expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787756587.673084:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x5b:0x0]: losing layout lock 00010000:00010000:2.0:1787756587.673097:0:15681:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8c0583595800 lock: 00000000311f3571/0x109d23f2ff214abd lrc: 1/0,0 mode: --/PR res: [0x280000401:0xf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b32536578a5af expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.673103:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbad7400/0x109d23f2ff2149c1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x5a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b32536578a355 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787756587.673124:0:15681:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8c0583595800 lock: 000000000a48c252/0x109d23f2ff214aa8 lrc: 1/0,0 mode: --/PR res: [0x280000401:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b32536578a585 expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787756587.673138:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x5a:0x0]: losing layout lock 00010000:00010000:3.0:1787756587.673159:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbad6600/0x109d23f2ff2149b3 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x77:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b32536578a332 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787756587.673160:0:15681:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8c0583595800 lock: 00000000c8e7e804/0x109d23f2ff214a9a lrc: 1/0,0 mode: --/PR res: [0x280000401:0x28:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b32536578a562 expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787756587.673200:0:15681:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8c0583595800 lock: 00000000a7a8a078/0x109d23f2ff214a8c lrc: 1/0,0 mode: --/PR res: [0x280000401:0x1e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b32536578a53f expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787756587.673209:0:15681:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8c0583595800 lock: 00000000eb892072/0x109d23f2ff214a70 lrc: 1/0,0 mode: --/PR res: [0x280000401:0x11:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b32536578a4f9 expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787756587.673212:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x77:0x0]: losing layout lock 00010000:00010000:2.0:1787756587.673226:0:15681:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8c0583595800 lock: 00000000a1031292/0x109d23f2ff214a3f lrc: 1/0,0 mode: --/PR res: [0x280000401:0x29:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b32536578a482 expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.673242:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbad6c00/0x109d23f2ff2149ac lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x73:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b32536578a316 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787756587.673259:0:15681:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8c0583595800 lock: 000000009ab4e9fe/0x109d23f2ff2149f2 lrc: 1/0,0 mode: --/PR res: [0x280000401:0x472:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b32536578a3c5 expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787756587.673285:0:15681:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8c0583595800 lock: 00000000993c1773/0x109d23f2ff2149c8 lrc: 1/0,0 mode: --/PR res: [0x280000401:0x25:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b32536578a35c expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.673304:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbab0400/0x109d23f2ff21499e lrc: 2/0,0 mode: PR/PR res: [0x200000400:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b32536578a2fa expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787756587.673320:0:15681:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8c0583595800 lock: 000000000a6856e8/0x109d23f2ff2149ba lrc: 1/0,0 mode: --/PR res: [0x280000401:0x34:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b32536578a339 expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.673337:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbab0000/0x109d23f2ff214997 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x73:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b32536578a2ec expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787756587.673362:0:15681:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8c0583595800 lock: 00000000de107905/0x109d23f2ff214974 lrc: 1/0,0 mode: --/PR res: [0x280000401:0x35:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b32536578a28a expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.673381:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbab1800/0x109d23f2ff214990 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x23:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b32536578a2d0 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787756587.673396:0:15681:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8c0583595800 lock: 0000000001e3b349/0x109d23f2ff21495f lrc: 1/0,0 mode: --/PR res: [0x280000401:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b32536578a24b expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787756587.673434:0:15681:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8c0583595800 lock: 0000000023450f2b/0x109d23f2ff214935 lrc: 1/0,0 mode: --/PR res: [0x280000401:0x12:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b32536578a1e2 expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.673436:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbab0c00/0x109d23f2ff214982 lrc: 2/0,0 mode: PR/PR res: [0x200000400:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b32536578a2b4 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787756587.673470:0:15681:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8c0583595800 lock: 00000000b4388321/0x109d23f2ff214927 lrc: 1/0,0 mode: --/PR res: [0x280000401:0x2b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b32536578a1bf expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787756587.673487:0:15681:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8c0583595800 lock: 0000000059ed1420/0x109d23f2ff214919 lrc: 1/0,0 mode: --/PR res: [0x280000401:0x24:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b32536578a19c expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.673489:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2d7b000/0x109d23f2ff21497b lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x23:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b32536578a2a6 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787756587.673526:0:15681:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8c0583595800 lock: 00000000b289b1c6/0x109d23f2ff2148fd lrc: 1/0,0 mode: --/PR res: [0x280000401:0x2c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b32536578a156 expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787756587.673556:0:15681:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8c0583595800 lock: 0000000099e050e6/0x109d23f2ff2148e1 lrc: 1/0,0 mode: --/PR res: [0x280000401:0x39:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b32536578a110 expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.673565:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbab1000/0x109d23f2ff21496d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b32536578a283 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787756587.673628:0:15681:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8c0583595800 lock: 00000000eeee5529/0x109d23f2ff214886 lrc: 1/0,0 mode: --/PR res: [0x280000401:0x5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b32536578a030 expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787756587.673644:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x7b:0x0]: losing layout lock 00010000:00010000:2.0:1787756587.673666:0:15681:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8c0583595800 lock: 000000001efae11a/0x109d23f2ff21480f lrc: 1/0,0 mode: --/PR res: [0x280000401:0x473:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b325365789f03 expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.673679:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2d7a600/0x109d23f2ff214966 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b32536578a267 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787756587.673688:0:15681:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8c0583595800 lock: 000000002e528c15/0x109d23f2ff21479f lrc: 1/0,0 mode: --/PR res: [0x280000401:0x15:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b325365789de4 expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787756587.673722:0:15681:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8c0583595800 lock: 000000001bf711a1/0x109d23f2ff214767 lrc: 1/0,0 mode: --/PR res: [0x280000401:0x26:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b325365789d51 expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.673727:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbab1e00/0x109d23f2ff214958 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b32536578a244 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787756587.673764:0:15681:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8c0583595800 lock: 00000000363d3556/0x109d23f2ff214759 lrc: 1/0,0 mode: --/PR res: [0x280000401:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b325365789d2e expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787756587.673764:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x9:0x0]: losing layout lock 00010000:00010000:3.0:1787756587.673789:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbba9a00/0x109d23f2ff21494a lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b32536578a221 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787756587.673792:0:15681:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 25 locks packed 00010000:00010000:2.0:1787756587.673818:0:15681:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8c0583595800 lock: 000000009f286827/0x109d23f2ff214bf1 lrc: 0/0,0 mode: --/PR res: [0x280000401:0x3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b32536578a8aa expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787756587.673826:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x3a:0x0]: losing layout lock 00010000:00010000:3.0:1787756587.673852:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbba9400/0x109d23f2ff21493c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x61:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b32536578a1fe expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787756587.673867:0:15681:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8c0583595800 lock: 0000000059bd96ce/0x109d23f2ff214bc0 lrc: 0/0,0 mode: --/PR res: [0x280000401:0x27:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b32536578a833 expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787756587.673919:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x61:0x0]: losing layout lock 00010000:00010000:3.0:1787756587.673940:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2d7b400/0x109d23f2ff21492e lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b32536578a1db expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787756587.673947:0:15681:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8c0583595800 lock: 000000006fe73ec1/0x109d23f2ff214ba4 lrc: 0/0,0 mode: --/PR res: [0x280000401:0x4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b32536578a7ed expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787756587.673962:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x34:0x0]: losing layout lock 00010000:00010000:3.0:1787756587.673980:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2d7ac00/0x109d23f2ff214920 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x66:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b32536578a1b8 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787756587.673993:0:15681:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8c0583595800 lock: 0000000080d18de9/0x109d23f2ff214b96 lrc: 0/0,0 mode: --/PR res: [0x280000401:0x2a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b32536578a7ca expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787756587.673997:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x66:0x0]: losing layout lock 00010000:00010000:3.0:1787756587.674013:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2d7a000/0x109d23f2ff214912 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x58:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b32536578a195 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787756587.674048:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x58:0x0]: losing layout lock 00010000:00010000:2.0:1787756587.674051:0:15681:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8c0583595800 lock: 00000000311f3571/0x109d23f2ff214abd lrc: 0/0,0 mode: --/PR res: [0x280000401:0xf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b32536578a5af expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.674064:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0585662a00/0x109d23f2ff214904 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x63:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b32536578a172 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787756587.674101:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x63:0x0]: losing layout lock 00010000:00010000:3.0:1787756587.674128:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbba8800/0x109d23f2ff2148f6 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x67:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b32536578a14f expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787756587.674134:0:15681:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8c0583595800 lock: 000000000a48c252/0x109d23f2ff214aa8 lrc: 0/0,0 mode: --/PR res: [0x280000401:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b32536578a585 expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787756587.674158:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x67:0x0]: losing layout lock 00010000:00010000:2.0:1787756587.674192:0:15681:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8c0583595800 lock: 00000000c8e7e804/0x109d23f2ff214a9a lrc: 0/0,0 mode: --/PR res: [0x280000401:0x28:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b32536578a562 expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.674192:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2d7b200/0x109d23f2ff2148e8 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b32536578a12c expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787756587.674246:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x6:0x0]: losing layout lock 00010000:00010000:2.0:1787756587.674247:0:15681:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8c0583595800 lock: 00000000a7a8a078/0x109d23f2ff214a8c lrc: 0/0,0 mode: --/PR res: [0x280000401:0x1e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b32536578a53f expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.674270:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbba8200/0x109d23f2ff2148da lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x2a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b32536578a109 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787756587.674286:0:15681:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8c0583595800 lock: 00000000eb892072/0x109d23f2ff214a70 lrc: 0/0,0 mode: --/PR res: [0x280000401:0x11:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b32536578a4f9 expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787756587.674312:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x2a:0x0]: losing layout lock 00010000:00010000:2.0:1787756587.674347:0:15681:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8c0583595800 lock: 00000000a1031292/0x109d23f2ff214a3f lrc: 0/0,0 mode: --/PR res: [0x280000401:0x29:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b32536578a482 expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.674352:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbab0a00/0x109d23f2ff2148d3 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x48:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b32536578a0ed expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787756587.674383:0:15681:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8c0583595800 lock: 000000009ab4e9fe/0x109d23f2ff2149f2 lrc: 0/0,0 mode: --/PR res: [0x280000401:0x472:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b32536578a3c5 expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787756587.674414:0:15681:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8c0583595800 lock: 00000000993c1773/0x109d23f2ff2149c8 lrc: 0/0,0 mode: --/PR res: [0x280000401:0x25:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b32536578a35c expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.674419:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbab0600/0x109d23f2ff2148c5 lrc: 2/0,0 mode: PR/PR res: [0x200000400:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b32536578a0d1 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787756587.674465:0:15681:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8c0583595800 lock: 000000000a6856e8/0x109d23f2ff2149ba lrc: 0/0,0 mode: --/PR res: [0x280000401:0x34:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b32536578a339 expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.674490:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbab0200/0x109d23f2ff2148be lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x48:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b32536578a0c3 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787756587.674536:0:15681:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8c0583595800 lock: 00000000de107905/0x109d23f2ff214974 lrc: 0/0,0 mode: --/PR res: [0x280000401:0x35:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b32536578a28a expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787756587.674584:0:15681:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8c0583595800 lock: 0000000001e3b349/0x109d23f2ff21495f lrc: 0/0,0 mode: --/PR res: [0x280000401:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b32536578a24b expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.674601:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbab1400/0x109d23f2ff2148b0 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b32536578a0a0 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787756587.674648:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x2e:0x0]: losing layout lock 00010000:00010000:2.0:1787756587.674655:0:15681:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8c0583595800 lock: 0000000023450f2b/0x109d23f2ff214935 lrc: 0/0,0 mode: --/PR res: [0x280000401:0x12:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b32536578a1e2 expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.674669:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0583073e00/0x109d23f2ff2148a2 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b32536578a076 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787756587.674692:0:15681:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8c0583595800 lock: 00000000b4388321/0x109d23f2ff214927 lrc: 0/0,0 mode: --/PR res: [0x280000401:0x2b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b32536578a1bf expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.674709:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0583073200/0x109d23f2ff21488d lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x4:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b32536578a04c expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787756587.674712:0:15681:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8c0583595800 lock: 0000000059ed1420/0x109d23f2ff214919 lrc: 0/0,0 mode: --/PR res: [0x280000401:0x24:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b32536578a19c expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.674748:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0583073800/0x109d23f2ff21487f lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b32536578a029 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787756587.674752:0:15681:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8c0583595800 lock: 00000000b289b1c6/0x109d23f2ff2148fd lrc: 0/0,0 mode: --/PR res: [0x280000401:0x2c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b32536578a156 expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787756587.674781:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x8:0x0]: losing layout lock 00010000:00010000:2.0:1787756587.674805:0:15681:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8c0583595800 lock: 0000000099e050e6/0x109d23f2ff2148e1 lrc: 0/0,0 mode: --/PR res: [0x280000401:0x39:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b32536578a110 expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.674812:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0583072600/0x109d23f2ff214878 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x78:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b32536578a00d expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787756587.674828:0:15681:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8c0583595800 lock: 00000000eeee5529/0x109d23f2ff214886 lrc: 0/0,0 mode: --/PR res: [0x280000401:0x5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b32536578a030 expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787756587.674854:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x78:0x0]: losing layout lock 00010000:00010000:2.0:1787756587.674869:0:15681:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8c0583595800 lock: 000000001efae11a/0x109d23f2ff21480f lrc: 0/0,0 mode: --/PR res: [0x280000401:0x473:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b325365789f03 expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.674922:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0583073a00/0x109d23f2ff214871 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x46:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b325365789ff1 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787756587.674923:0:15681:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8c0583595800 lock: 000000002e528c15/0x109d23f2ff21479f lrc: 0/0,0 mode: --/PR res: [0x280000401:0x15:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b325365789de4 expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787756587.674980:0:15681:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8c0583595800 lock: 000000001bf711a1/0x109d23f2ff214767 lrc: 0/0,0 mode: --/PR res: [0x280000401:0x26:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b325365789d51 expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.674984:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0583072800/0x109d23f2ff214863 lrc: 2/0,0 mode: PR/PR res: [0x200000400:0x9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b325365789fd5 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787756587.675007:0:15681:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8c0583595800 lock: 00000000363d3556/0x109d23f2ff214759 lrc: 0/0,0 mode: --/PR res: [0x280000401:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9b2b325365789d2e expref: -99 pid: 232705 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.675032:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0583072400/0x109d23f2ff21485c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x46:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b325365789fc7 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000100:00100000:2.0:1787756587.675050:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00010000:00010000:3.0:1787756587.675066:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0583072c00/0x109d23f2ff21484e lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b325365789f9d expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000100:00080000:2.0:1787756587.675071:0:2401:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff8c05b2e71f80 x1874588654754432/t0(0) o103->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:17/18 lens 512/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00010000:00010000:3.0:1787756587.675083:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0583073c00/0x109d23f2ff214839 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x6:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b325365789f73 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.675124:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b303c600/0x109d23f2ff21482b lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b325365789f49 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.675169:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b303d200/0x109d23f2ff214816 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x2a:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b325365789f1f expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.675213:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0585663e00/0x109d23f2ff214808 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x167:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b325365789efc expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787756587.675240:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x167:0x0]: losing layout lock 00010000:00010000:3.0:1787756587.675283:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b303da00/0x109d23f2ff2147fa lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b325365789ed9 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787756587.675326:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x2c:0x0]: losing layout lock 00010000:00010000:2.0:1787756587.675344:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756587.675351:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756587.675357:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756587.675357:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b303c200/0x109d23f2ff2147f3 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x88:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b325365789ebd expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.675405:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0585663c00/0x109d23f2ff2147ec lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x88:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b325365789eaf expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.675454:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0585663000/0x109d23f2ff2147de lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x5f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b325365789e8c expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787756587.675469:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x5f:0x0]: losing layout lock 00010000:00010000:3.0:1787756587.675482:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0585662000/0x109d23f2ff2147d0 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x166:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b325365789e69 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787756587.675489:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x166:0x0]: losing layout lock 00010000:00010000:3.0:1787756587.675497:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0585662400/0x109d23f2ff2147c2 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b325365789e46 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787756587.675509:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x1b:0x0]: losing layout lock 00010000:00010000:3.0:1787756587.675517:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0585663600/0x109d23f2ff2147b4 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x161:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b325365789e23 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787756587.675523:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x161:0x0]: losing layout lock 00010000:00010000:3.0:1787756587.675530:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b303c000/0x109d23f2ff2147a6 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b325365789e00 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787756587.675537:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x33:0x0]: losing layout lock 00010000:00010000:3.0:1787756587.675546:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0585662c00/0x109d23f2ff214798 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b325365789ddd expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787756587.675553:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x39:0x0]: losing layout lock 00010000:00010000:3.0:1787756587.675561:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b3220000/0x109d23f2ff214791 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x447:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b325365789dc1 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.675613:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b3221200/0x109d23f2ff21478a lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x447:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b325365789db3 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.675638:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b3220c00/0x109d23f2ff214783 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1f:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b325365789d97 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.675656:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b3220800/0x109d23f2ff214775 lrc: 2/0,0 mode: PR/PR res: [0x200000400:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b325365789d7b expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.675667:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2d7aa00/0x109d23f2ff21476e lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1f:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b325365789d6d expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.675685:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2d7ae00/0x109d23f2ff214760 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x5c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b325365789d4a expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787756587.675693:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x5c:0x0]: losing layout lock 00010000:00010000:3.0:1787756587.675702:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2d7b800/0x109d23f2ff214752 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b325365789d27 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787756587.675709:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x2:0x0]: losing layout lock 00010000:00010000:3.0:1787756587.675717:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b3221a00/0x109d23f2ff214744 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9b2b325365789cfd expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787756587.675724:0:34913:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x5:0x0]: losing layout lock 00010000:00010000:3.0:1787756587.675732:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b3221800/0x109d23f2ff21473d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x21:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b325365789ce1 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.675745:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b3221600/0x109d23f2ff21472f lrc: 2/0,0 mode: PR/PR res: [0x200000400:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b325365789cc5 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.675753:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b3220200/0x109d23f2ff214728 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x21:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b325365789cb7 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.675765:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b3220e00/0x109d23f2ff214721 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x29:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b325365789c9b expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.675776:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b3221c00/0x109d23f2ff21471a lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x29:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b325365789c8d expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.675789:0:34913:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b3221000/0x109d23f2ff214713 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9b2b325365789c71 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.675812:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2e9dc00/0x109d23f2ff214bf8 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x83:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a8c6 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.675822:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2e9c400/0x109d23f2ff214bea lrc: 1/0,0 mode: --/PR res: [0x200000402:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a8a3 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.675831:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2e9c200/0x109d23f2ff214bdc lrc: 1/0,0 mode: --/PR res: [0x200000401:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a879 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.675841:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0591285000/0x109d23f2ff214bc7 lrc: 1/0,0 mode: --/PR res: [0x240000402:0x15:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a84f expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.675851:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b3363600/0x109d23f2ff214bb9 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x5e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a82c expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.675861:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbba9e00/0x109d23f2ff214bab lrc: 1/0,0 mode: --/PR res: [0x200000402:0x65:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a809 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.675871:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2d7be00/0x109d23f2ff214b9d lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a7e6 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.675899:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2d7b600/0x109d23f2ff214b8f lrc: 1/0,0 mode: --/PR res: [0x200000402:0x64:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a7c3 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.675906:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2e9da00/0x109d23f2ff214b81 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x80:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a7a0 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.675913:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2e9d600/0x109d23f2ff214b7a lrc: 1/0,0 mode: --/PR res: [0x200000402:0x162:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a784 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.675920:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0592633800/0x109d23f2ff214b6c lrc: 1/0,0 mode: --/PR res: [0x200000bd0:0x6b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a768 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.675927:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0592633a00/0x109d23f2ff214b65 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x162:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a75a expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.675932:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0592633400/0x109d23f2ff214b5e lrc: 1/0,0 mode: --/PR res: [0x200000402:0x22:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a73e expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.675937:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0592632200/0x109d23f2ff214b50 lrc: 1/0,0 mode: --/PR res: [0x200000400:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a722 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.675943:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0592632e00/0x109d23f2ff214b49 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x22:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a714 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.675948:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0592632000/0x109d23f2ff214b3b lrc: 1/0,0 mode: --/PR res: [0x200000402:0x59:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a6f1 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.675953:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0592633000/0x109d23f2ff214b2d lrc: 1/0,0 mode: --/PR res: [0x200000402:0x57:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a6ce expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.675958:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0592633200/0x109d23f2ff214b26 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x7c:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a6b2 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.675963:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0592632600/0x109d23f2ff214b18 lrc: 1/0,0 mode: --/PR res: [0x200000400:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a696 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.675968:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0592632400/0x109d23f2ff214b11 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x7c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a688 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.675975:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0592633600/0x109d23f2ff214b0a lrc: 1/0,0 mode: --/PR res: [0x200000402:0x20:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a66c expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.675981:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0589f74a00/0x109d23f2ff214afc lrc: 1/0,0 mode: --/PR res: [0x200000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a650 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.675986:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0589f75000/0x109d23f2ff214af5 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x20:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a642 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.675991:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0589f74e00/0x109d23f2ff214ae7 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x168:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a61f expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.675997:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0589f75a00/0x109d23f2ff214ad9 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a5f5 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676002:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0589f74400/0x109d23f2ff214ac4 lrc: 1/0,0 mode: --/PR res: [0x240000402:0x16:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a5cb expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.676007:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0589f75c00/0x109d23f2ff214ab6 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a5a8 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676012:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0589f74800/0x109d23f2ff214aa1 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a57e expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676018:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2d7a200/0x109d23f2ff214a93 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x60:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a55b expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676023:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbace400/0x109d23f2ff214a85 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a538 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676029:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbacf400/0x109d23f2ff214a77 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x26:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a515 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676034:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbacf000/0x109d23f2ff214a62 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a4eb expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676039:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbacea00/0x109d23f2ff214a4d lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1b79:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a4ac expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.676045:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbacf800/0x109d23f2ff214a46 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1b79:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a49e expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676050:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbacfa00/0x109d23f2ff214a38 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x62:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a47b expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676055:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbacfc00/0x109d23f2ff214a2a lrc: 1/0,0 mode: --/PR res: [0x200000402:0x5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a458 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676060:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbacec00/0x109d23f2ff214a1c lrc: 1/0,0 mode: --/PR res: [0x200000401:0x9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a42e expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676066:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbad7c00/0x109d23f2ff214a07 lrc: 1/0,0 mode: --/PR res: [0x240000402:0x1e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a404 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.676071:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbad6200/0x109d23f2ff2149f9 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x79:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a3e1 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676076:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbad7a00/0x109d23f2ff2149eb lrc: 1/0,0 mode: --/PR res: [0x200000402:0x164:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a3be expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676082:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbad7600/0x109d23f2ff2149dd lrc: 1/0,0 mode: --/PR res: [0x200000402:0x76:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a39b expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676087:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbad6a00/0x109d23f2ff2149cf lrc: 1/0,0 mode: --/PR res: [0x200000402:0x5b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a378 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676092:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbad7400/0x109d23f2ff2149c1 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x5a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a355 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676098:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbad6600/0x109d23f2ff2149b3 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x77:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a332 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676103:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbad6c00/0x109d23f2ff2149ac lrc: 1/0,0 mode: --/PR res: [0x200000402:0x73:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a316 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.676108:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbab0400/0x109d23f2ff21499e lrc: 1/0,0 mode: --/PR res: [0x200000400:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a2fa expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676113:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbab0000/0x109d23f2ff214997 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x73:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a2ec expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676119:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbab1800/0x109d23f2ff214990 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x23:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a2d0 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.676124:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbab0c00/0x109d23f2ff214982 lrc: 1/0,0 mode: --/PR res: [0x200000400:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a2b4 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676129:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2d7b000/0x109d23f2ff21497b lrc: 1/0,0 mode: --/PR res: [0x200000402:0x23:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a2a6 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676135:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbab1000/0x109d23f2ff21496d lrc: 1/0,0 mode: --/PR res: [0x200000402:0x7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a283 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676141:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2d7a600/0x109d23f2ff214966 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a267 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676146:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbab1e00/0x109d23f2ff214958 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a244 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676152:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbba9a00/0x109d23f2ff21494a lrc: 1/0,0 mode: --/PR res: [0x200000402:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a221 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676157:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbba9400/0x109d23f2ff21493c lrc: 1/0,0 mode: --/PR res: [0x200000402:0x61:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a1fe expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676162:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2d7b400/0x109d23f2ff21492e lrc: 1/0,0 mode: --/PR res: [0x200000402:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a1db expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676168:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2d7ac00/0x109d23f2ff214920 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x66:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a1b8 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676173:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2d7a000/0x109d23f2ff214912 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x58:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a195 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676178:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0585662a00/0x109d23f2ff214904 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x63:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a172 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676186:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbba8800/0x109d23f2ff2148f6 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x67:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a14f expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676196:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2d7b200/0x109d23f2ff2148e8 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a12c expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676205:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbba8200/0x109d23f2ff2148da lrc: 1/0,0 mode: --/PR res: [0x200000403:0x2a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a109 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676215:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbab0a00/0x109d23f2ff2148d3 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x48:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a0ed expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.676225:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbab0600/0x109d23f2ff2148c5 lrc: 1/0,0 mode: --/PR res: [0x200000400:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a0d1 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676235:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbab0200/0x109d23f2ff2148be lrc: 1/0,0 mode: --/PR res: [0x200000402:0x48:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a0c3 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676244:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbab1400/0x109d23f2ff2148b0 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a0a0 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676250:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0583073e00/0x109d23f2ff2148a2 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a076 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676255:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0583073200/0x109d23f2ff21488d lrc: 1/0,0 mode: --/PR res: [0x240000402:0x4:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a04c expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.676260:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0583073800/0x109d23f2ff21487f lrc: 1/0,0 mode: --/PR res: [0x200000402:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a029 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676266:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0583072600/0x109d23f2ff214878 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x78:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a00d expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676272:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0583073a00/0x109d23f2ff214871 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x46:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b325365789ff1 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.676281:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0583072800/0x109d23f2ff214863 lrc: 1/0,0 mode: --/PR res: [0x200000400:0x9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b325365789fd5 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676290:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0583072400/0x109d23f2ff21485c lrc: 1/0,0 mode: --/PR res: [0x200000402:0x46:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b325365789fc7 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676299:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0583072c00/0x109d23f2ff21484e lrc: 1/0,0 mode: --/PR res: [0x200000401:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b325365789f9d expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676309:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0583073c00/0x109d23f2ff214839 lrc: 1/0,0 mode: --/PR res: [0x240000402:0x6:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b325365789f73 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.676320:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b303c600/0x109d23f2ff21482b lrc: 1/0,0 mode: --/PR res: [0x200000401:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b325365789f49 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676329:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b303d200/0x109d23f2ff214816 lrc: 1/0,0 mode: --/PR res: [0x240000402:0x2a:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b325365789f1f expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.676335:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0585663e00/0x109d23f2ff214808 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x167:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b325365789efc expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676342:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b303da00/0x109d23f2ff2147fa lrc: 1/0,0 mode: --/PR res: [0x200000402:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b325365789ed9 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676347:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b303c200/0x109d23f2ff2147f3 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x88:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b325365789ebd expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.676353:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0585663c00/0x109d23f2ff2147ec lrc: 1/0,0 mode: --/PR res: [0x200000402:0x88:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b325365789eaf expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676359:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0585663000/0x109d23f2ff2147de lrc: 1/0,0 mode: --/PR res: [0x200000402:0x5f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b325365789e8c expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676364:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0585662000/0x109d23f2ff2147d0 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x166:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b325365789e69 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676369:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0585662400/0x109d23f2ff2147c2 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b325365789e46 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676374:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0585663600/0x109d23f2ff2147b4 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x161:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b325365789e23 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676380:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b303c000/0x109d23f2ff2147a6 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b325365789e00 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676385:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0585662c00/0x109d23f2ff214798 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b325365789ddd expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676391:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b3220000/0x109d23f2ff214791 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x447:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b325365789dc1 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.676400:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b3221200/0x109d23f2ff21478a lrc: 1/0,0 mode: --/PR res: [0x200000402:0x447:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b325365789db3 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676408:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b3220c00/0x109d23f2ff214783 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x1f:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b325365789d97 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.676419:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b3220800/0x109d23f2ff214775 lrc: 1/0,0 mode: --/PR res: [0x200000400:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b325365789d7b expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676429:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2d7aa00/0x109d23f2ff21476e lrc: 1/0,0 mode: --/PR res: [0x200000402:0x1f:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b325365789d6d expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676440:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2d7ae00/0x109d23f2ff214760 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x5c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b325365789d4a expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676448:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2d7b800/0x109d23f2ff214752 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b325365789d27 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676454:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b3221a00/0x109d23f2ff214744 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b325365789cfd expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676459:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b3221800/0x109d23f2ff21473d lrc: 1/0,0 mode: --/PR res: [0x200000402:0x21:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b325365789ce1 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.676464:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b3221600/0x109d23f2ff21472f lrc: 1/0,0 mode: --/PR res: [0x200000400:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b325365789cc5 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676470:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b3220200/0x109d23f2ff214728 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x21:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b325365789cb7 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676475:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b3220e00/0x109d23f2ff214721 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x29:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b325365789c9b expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.676480:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b3221c00/0x109d23f2ff21471a lrc: 1/0,0 mode: --/PR res: [0x200000403:0x29:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b325365789c8d expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676487:0:34913:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b3221000/0x109d23f2ff214713 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b325365789c71 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.676492:0:34913:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 101 locks packed 00010000:00010000:3.0:1787756587.676516:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2e9dc00/0x109d23f2ff214bf8 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x83:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a8c6 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676530:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2e9c400/0x109d23f2ff214bea lrc: 0/0,0 mode: --/PR res: [0x200000402:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a8a3 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676546:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2e9c200/0x109d23f2ff214bdc lrc: 0/0,0 mode: --/PR res: [0x200000401:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a879 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000100:00080000:2.0:1787756587.676557:0:2401:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff8c05a9f78000 x1874588654754816/t0(0) o103->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:17/18 lens 1120/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00010000:00010000:3.0:1787756587.676559:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0591285000/0x109d23f2ff214bc7 lrc: 0/0,0 mode: --/PR res: [0x240000402:0x15:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a84f expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.676615:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b3363600/0x109d23f2ff214bb9 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x5e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a82c expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676638:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbba9e00/0x109d23f2ff214bab lrc: 0/0,0 mode: --/PR res: [0x200000402:0x65:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a809 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676650:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2d7be00/0x109d23f2ff214b9d lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a7e6 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676659:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2d7b600/0x109d23f2ff214b8f lrc: 0/0,0 mode: --/PR res: [0x200000402:0x64:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a7c3 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676674:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2e9da00/0x109d23f2ff214b81 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x80:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a7a0 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676682:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2e9d600/0x109d23f2ff214b7a lrc: 0/0,0 mode: --/PR res: [0x200000402:0x162:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a784 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.676689:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0592633800/0x109d23f2ff214b6c lrc: 0/0,0 mode: --/PR res: [0x200000bd0:0x6b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a768 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676697:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0592633a00/0x109d23f2ff214b65 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x162:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a75a expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676705:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0592633400/0x109d23f2ff214b5e lrc: 0/0,0 mode: --/PR res: [0x200000402:0x22:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a73e expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.676711:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0592632200/0x109d23f2ff214b50 lrc: 0/0,0 mode: --/PR res: [0x200000400:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a722 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676719:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0592632e00/0x109d23f2ff214b49 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x22:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a714 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676727:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0592632000/0x109d23f2ff214b3b lrc: 0/0,0 mode: --/PR res: [0x200000402:0x59:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a6f1 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676735:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0592633000/0x109d23f2ff214b2d lrc: 0/0,0 mode: --/PR res: [0x200000402:0x57:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a6ce expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676748:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0592633200/0x109d23f2ff214b26 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x7c:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a6b2 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.676754:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0592632600/0x109d23f2ff214b18 lrc: 0/0,0 mode: --/PR res: [0x200000400:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a696 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676761:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0592632400/0x109d23f2ff214b11 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x7c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a688 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676769:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0592633600/0x109d23f2ff214b0a lrc: 0/0,0 mode: --/PR res: [0x200000402:0x20:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a66c expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.676775:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0589f74a00/0x109d23f2ff214afc lrc: 0/0,0 mode: --/PR res: [0x200000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a650 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676782:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0589f75000/0x109d23f2ff214af5 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x20:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a642 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676790:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0589f74e00/0x109d23f2ff214ae7 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x168:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a61f expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676798:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0589f75a00/0x109d23f2ff214ad9 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a5f5 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676805:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0589f74400/0x109d23f2ff214ac4 lrc: 0/0,0 mode: --/PR res: [0x240000402:0x16:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a5cb expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.676822:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0589f75c00/0x109d23f2ff214ab6 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a5a8 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676837:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0589f74800/0x109d23f2ff214aa1 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a57e expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676851:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2d7a200/0x109d23f2ff214a93 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x60:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a55b expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676866:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbace400/0x109d23f2ff214a85 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a538 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676897:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbacf400/0x109d23f2ff214a77 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x26:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a515 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676911:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbacf000/0x109d23f2ff214a62 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a4eb expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676922:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbacea00/0x109d23f2ff214a4d lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1b79:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a4ac expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.676930:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbacf800/0x109d23f2ff214a46 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1b79:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a49e expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676938:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbacfa00/0x109d23f2ff214a38 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x62:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a47b expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676951:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbacfc00/0x109d23f2ff214a2a lrc: 0/0,0 mode: --/PR res: [0x200000402:0x5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a458 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676960:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbacec00/0x109d23f2ff214a1c lrc: 0/0,0 mode: --/PR res: [0x200000401:0x9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a42e expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676967:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbad7c00/0x109d23f2ff214a07 lrc: 0/0,0 mode: --/PR res: [0x240000402:0x1e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a404 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.676975:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbad6200/0x109d23f2ff2149f9 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x79:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a3e1 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676983:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbad7a00/0x109d23f2ff2149eb lrc: 0/0,0 mode: --/PR res: [0x200000402:0x164:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a3be expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676991:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbad7600/0x109d23f2ff2149dd lrc: 0/0,0 mode: --/PR res: [0x200000402:0x76:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a39b expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.676999:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbad6a00/0x109d23f2ff2149cf lrc: 0/0,0 mode: --/PR res: [0x200000402:0x5b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a378 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677007:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbad7400/0x109d23f2ff2149c1 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x5a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a355 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677015:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbad6600/0x109d23f2ff2149b3 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x77:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a332 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677028:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbad6c00/0x109d23f2ff2149ac lrc: 0/0,0 mode: --/PR res: [0x200000402:0x73:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a316 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.677034:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbab0400/0x109d23f2ff21499e lrc: 0/0,0 mode: --/PR res: [0x200000400:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a2fa expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677042:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbab0000/0x109d23f2ff214997 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x73:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a2ec expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677049:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbab1800/0x109d23f2ff214990 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x23:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a2d0 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.677055:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbab0c00/0x109d23f2ff214982 lrc: 0/0,0 mode: --/PR res: [0x200000400:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a2b4 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677063:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2d7b000/0x109d23f2ff21497b lrc: 0/0,0 mode: --/PR res: [0x200000402:0x23:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a2a6 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677070:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbab1000/0x109d23f2ff21496d lrc: 0/0,0 mode: --/PR res: [0x200000402:0x7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a283 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677079:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2d7a600/0x109d23f2ff214966 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a267 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677087:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbab1e00/0x109d23f2ff214958 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a244 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677100:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbba9a00/0x109d23f2ff21494a lrc: 0/0,0 mode: --/PR res: [0x200000402:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a221 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677109:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbba9400/0x109d23f2ff21493c lrc: 0/0,0 mode: --/PR res: [0x200000402:0x61:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a1fe expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677117:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2d7b400/0x109d23f2ff21492e lrc: 0/0,0 mode: --/PR res: [0x200000402:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a1db expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677125:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2d7ac00/0x109d23f2ff214920 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x66:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a1b8 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677134:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2d7a000/0x109d23f2ff214912 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x58:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a195 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677142:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0585662a00/0x109d23f2ff214904 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x63:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a172 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677149:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbba8800/0x109d23f2ff2148f6 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x67:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a14f expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677157:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2d7b200/0x109d23f2ff2148e8 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a12c expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677165:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbba8200/0x109d23f2ff2148da lrc: 0/0,0 mode: --/PR res: [0x200000403:0x2a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a109 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677176:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbab0a00/0x109d23f2ff2148d3 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x48:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a0ed expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.677186:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbab0600/0x109d23f2ff2148c5 lrc: 0/0,0 mode: --/PR res: [0x200000400:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a0d1 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677200:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbab0200/0x109d23f2ff2148be lrc: 0/0,0 mode: --/PR res: [0x200000402:0x48:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a0c3 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677214:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05bbab1400/0x109d23f2ff2148b0 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a0a0 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677229:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0583073e00/0x109d23f2ff2148a2 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a076 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677242:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0583073200/0x109d23f2ff21488d lrc: 0/0,0 mode: --/PR res: [0x240000402:0x4:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b32536578a04c expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.677252:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0583073800/0x109d23f2ff21487f lrc: 0/0,0 mode: --/PR res: [0x200000402:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a029 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677260:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0583072600/0x109d23f2ff214878 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x78:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b32536578a00d expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677268:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0583073a00/0x109d23f2ff214871 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x46:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b325365789ff1 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.677282:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0583072800/0x109d23f2ff214863 lrc: 0/0,0 mode: --/PR res: [0x200000400:0x9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b325365789fd5 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677296:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0583072400/0x109d23f2ff21485c lrc: 0/0,0 mode: --/PR res: [0x200000402:0x46:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b325365789fc7 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677309:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0583072c00/0x109d23f2ff21484e lrc: 0/0,0 mode: --/PR res: [0x200000401:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b325365789f9d expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677323:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0583073c00/0x109d23f2ff214839 lrc: 0/0,0 mode: --/PR res: [0x240000402:0x6:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b325365789f73 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.677336:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b303c600/0x109d23f2ff21482b lrc: 0/0,0 mode: --/PR res: [0x200000401:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b325365789f49 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677345:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b303d200/0x109d23f2ff214816 lrc: 0/0,0 mode: --/PR res: [0x240000402:0x2a:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b325365789f1f expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.677354:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0585663e00/0x109d23f2ff214808 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x167:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b325365789efc expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677362:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b303da00/0x109d23f2ff2147fa lrc: 0/0,0 mode: --/PR res: [0x200000402:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b325365789ed9 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677371:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b303c200/0x109d23f2ff2147f3 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x88:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b325365789ebd expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.677382:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0585663c00/0x109d23f2ff2147ec lrc: 0/0,0 mode: --/PR res: [0x200000402:0x88:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b325365789eaf expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677390:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0585663000/0x109d23f2ff2147de lrc: 0/0,0 mode: --/PR res: [0x200000402:0x5f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b325365789e8c expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677403:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0585662000/0x109d23f2ff2147d0 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x166:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b325365789e69 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677416:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0585662400/0x109d23f2ff2147c2 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b325365789e46 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677430:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0585663600/0x109d23f2ff2147b4 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x161:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b325365789e23 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677445:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b303c000/0x109d23f2ff2147a6 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b325365789e00 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677456:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c0585662c00/0x109d23f2ff214798 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b325365789ddd expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677465:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b3220000/0x109d23f2ff214791 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x447:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b325365789dc1 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.677472:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b3221200/0x109d23f2ff21478a lrc: 0/0,0 mode: --/PR res: [0x200000402:0x447:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b325365789db3 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677484:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b3220c00/0x109d23f2ff214783 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x1f:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b325365789d97 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.677491:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b3220800/0x109d23f2ff214775 lrc: 0/0,0 mode: --/PR res: [0x200000400:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b325365789d7b expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677498:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2d7aa00/0x109d23f2ff21476e lrc: 0/0,0 mode: --/PR res: [0x200000402:0x1f:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b325365789d6d expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677506:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2d7ae00/0x109d23f2ff214760 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x5c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b325365789d4a expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677515:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b2d7b800/0x109d23f2ff214752 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b325365789d27 expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677523:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b3221a00/0x109d23f2ff214744 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9b2b325365789cfd expref: -99 pid: 232705 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677531:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b3221800/0x109d23f2ff21473d lrc: 0/0,0 mode: --/PR res: [0x200000402:0x21:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b325365789ce1 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.677537:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b3221600/0x109d23f2ff21472f lrc: 0/0,0 mode: --/PR res: [0x200000400:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b325365789cc5 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677545:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b3220200/0x109d23f2ff214728 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x21:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b325365789cb7 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677557:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b3220e00/0x109d23f2ff214721 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x29:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b325365789c9b expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.677563:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b3221c00/0x109d23f2ff21471a lrc: 0/0,0 mode: --/PR res: [0x200000403:0x29:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b325365789c8d expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787756587.677587:0:34913:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8c0583595800 lock: ffff8c05b3221000/0x109d23f2ff214713 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9b2b325365789c71 expref: -99 pid: 232705 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787756587.677603:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756587.677606:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756587.677609:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756589.718050:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787756592.214357:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654754944:192.168.201.107@tcp:250:kworker.0 00000800:00000100:3.0:1787756592.214575:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756592.214776:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756592/real 1787756592] req@ffff8c05b2d98380 x1874588654754944/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756592.214825:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654754944/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756592.214842:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756592.214849:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756592.214980:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654754944:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787756592.215013:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654755072:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787756592.215054:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756592.215075:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756592/real 1787756592] req@ffff8c05b2d9a300 x1874588654755072/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756592.215091:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588654755072/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756592.215101:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756592.215105:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756592.215109:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654755072:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756592.215137:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654755200:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787756592.215159:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756592.215171:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756592/real 1787756592] req@ffff8c05b2d98700 x1874588654755200/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756592.215185:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654755200/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756592.215194:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756592.215198:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756592.215202:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654755200:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756592.215208:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654755328:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787756592.215227:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756592.215238:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756592/real 1787756592] req@ffff8c05b2d9b480 x1874588654755328/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756592.215251:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654755328/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756592.215261:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756592.215264:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756592.215268:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654755328:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787756592.215274:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654755456:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787756592.215291:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756592.215301:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756592/real 1787756592] req@ffff8c05b2d9b100 x1874588654755456/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756592.215314:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654755456/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756592.215323:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756592.215327:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756592.215330:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654755456:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756592.215336:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654755584:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787756592.215352:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756592.215361:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756592/real 1787756592] req@ffff8c05b2d9aa00 x1874588654755584/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756592.215374:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588654755584/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756592.215383:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756592.215386:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756592.215391:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654755584:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756592.215397:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654755712:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787756592.215414:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756592.215423:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756592/real 1787756592] req@ffff8c05b2d98000 x1874588654755712/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756592.215436:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654755712/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756592.215445:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756592.215448:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756592.215451:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654755712:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787756592.215457:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654755840:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787756592.215473:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756592.215482:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756592/real 1787756592] req@ffff8c05b2d99880 x1874588654755840/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756592.215529:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654755840/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756592.215540:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756592.215543:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756592.215547:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654755840:192.168.201.107@tcp:8:kworker.0 00010000:00010000:2.0:1787756593.814113:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756595.862042:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756597.334072:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756597.334106:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787756597.334111:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756597.334120:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9653 00000100:00080000:3.0:1787756597.334139:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756597.334145:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787756597.334179:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588654755968/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756597.334205:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756597.334208:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756597.334210:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756597.334213:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9653 00000100:00080000:3.0:1787756597.334216:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756597.334219:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756597.334226:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756597.334235:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588654756096/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756597.334244:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756597.334247:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756597.334249:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756597.334254:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9653 00000100:00080000:3.0:1787756597.334261:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756597.334265:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756597.334268:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756597.334276:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588654756224/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756597.334301:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756597.334305:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756597.334307:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756597.334310:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9653 00000100:00080000:3.0:1787756597.334316:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756597.334320:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756597.334324:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756597.334334:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588654756352/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756597.334343:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756597.334346:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756597.334347:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756597.334350:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9653 00000100:00080000:3.0:1787756597.334353:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756597.334356:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756597.334370:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756597.334385:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79c00 x1874588654756480/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756597.334393:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756597.334396:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756597.334417:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756597.334422:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9653 00000100:00080000:3.0:1787756597.334428:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756597.334449:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756597.334454:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756597.334467:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588654756608/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756597.334487:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756597.334490:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787756597.334492:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756597.334495:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9653 00000100:00080000:3.0:1787756597.334498:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756597.334503:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756597.334510:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756597.334522:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588654756736/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756597.334533:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756597.334539:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756597.334543:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756597.334550:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9653 00000100:00080000:3.0:1787756597.334553:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756597.334558:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756597.334565:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756597.334572:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588654756864/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787756597.910035:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756602.006103:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756602.454008:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756602.454022:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787756602.454028:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756602.454041:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9658 00000100:00080000:3.0:1787756602.454054:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756602.454064:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787756602.454112:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588654756992/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756602.454151:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756602.454156:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756602.454159:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756602.454173:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9658 00000100:00080000:3.0:1787756602.454177:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756602.454182:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756602.454194:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756602.454213:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588654757120/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756602.454228:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756602.454235:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756602.454240:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756602.454248:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9658 00000100:00080000:3.0:1787756602.454255:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756602.454264:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756602.454271:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756602.454286:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588654757248/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756602.454306:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756602.454334:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756602.454338:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756602.454347:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9658 00000100:00080000:3.0:1787756602.454356:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756602.454364:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756602.454383:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756602.454415:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588654757376/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756602.454446:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756602.454465:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756602.454469:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756602.454485:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9658 00000100:00080000:3.0:1787756602.454500:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756602.454517:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756602.454528:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756602.454558:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79c00 x1874588654757504/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756602.454574:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756602.454578:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756602.454586:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756602.454604:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9658 00000100:00080000:3.0:1787756602.454613:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756602.454622:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756602.454632:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756602.454661:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588654757632/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756602.454693:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756602.454710:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787756602.454715:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756602.454721:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9658 00000100:00080000:3.0:1787756602.454728:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756602.454739:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756602.454748:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756602.454770:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588654757760/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756602.454806:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756602.454823:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756602.454826:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756602.454837:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9658 00000100:00080000:3.0:1787756602.454848:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756602.454853:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756602.454860:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756602.454917:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588654757888/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787756604.054131:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756606.102037:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787756607.574379:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654758016:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787756607.574580:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654758144:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756607.574718:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654758272:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756607.574798:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654758400:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787756607.574917:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654758528:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756607.574996:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654758656:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756607.575089:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654758784:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787756607.575156:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654758912:192.168.201.107@tcp:8:kworker.0 00010000:00010000:2.0:1787756610.198064:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787756610.646442:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756607/real 1787756610] req@ffff8c05b2d9bb80 x1874588654758144/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756610.646488:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588654758144/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756610.646510:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756610.646519:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756610.646526:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654758144:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756610.646537:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756607/real 1787756610] req@ffff8c05b2d99880 x1874588654758272/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756610.646555:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654758272/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756610.646565:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756610.646570:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756610.646574:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654758272:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756610.646583:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756607/real 1787756610] req@ffff8c05b2d98000 x1874588654758400/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756610.646599:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654758400/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756610.646609:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756610.646614:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756610.646618:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654758400:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787756610.646627:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756607/real 1787756610] req@ffff8c05b2d99f80 x1874588654758016/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756610.646642:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654758016/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756610.646652:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756610.646657:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756610.646661:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654758016:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787756610.646704:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756607/real 1787756610] req@ffff8c05b2d98700 x1874588654758912/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756610.646738:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654758912/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756610.646748:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756610.646753:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756610.646758:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654758912:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787756610.646766:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756607/real 1787756610] req@ffff8c05b2d9aa00 x1874588654758528/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756610.646782:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588654758528/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756610.646793:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756610.646797:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756610.646801:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654758528:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756610.646810:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756607/real 1787756610] req@ffff8c05b2d9b100 x1874588654758656/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756610.646824:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654758656/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756610.646833:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756610.646837:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756610.646841:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654758656:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756610.646849:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756607/real 1787756610] req@ffff8c05b2d9b480 x1874588654758784/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756610.646863:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654758784/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756610.646872:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756610.646875:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756610.646879:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654758784:192.168.201.107@tcp:8:kworker.0 00010000:00010000:3.0:1787756612.246012:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787756612.694351:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654759040:192.168.201.107@tcp:250:kworker.0 00000800:00000100:3.0:1787756612.694478:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756612.694566:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756612/real 1787756612] req@ffff8c05a9f76680 x1874588654759040/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756612.694635:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588654759040/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756612.694676:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756612.694691:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756612.694701:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654759040:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787756612.694772:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654759168:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787756612.694848:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756612.694944:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756612/real 1787756612] req@ffff8c05a9f77b80 x1874588654759168/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756612.695005:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588654759168/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756612.695064:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756612.695081:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756612.695100:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654759168:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756612.695156:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654759296:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787756612.695213:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756612.695244:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756612/real 1787756612] req@ffff8c05a9f76d80 x1874588654759296/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756612.695273:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588654759296/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756612.695290:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756612.695297:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756612.695305:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654759296:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756612.695319:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654759424:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787756612.695364:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756612.695389:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756612/real 1787756612] req@ffff8c05a9f76a00 x1874588654759424/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756612.695415:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588654759424/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756612.695433:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756612.695439:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756612.695447:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654759424:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787756612.695461:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654759552:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787756612.695504:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756612.695529:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756612/real 1787756612] req@ffff8c05a9f74380 x1874588654759552/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756612.695554:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588654759552/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756612.695719:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756612.695727:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756612.695734:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654759552:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756612.695772:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654759680:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787756612.695823:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756612.695849:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756612/real 1787756612] req@ffff8c05a9f75c00 x1874588654759680/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756612.695912:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588654759680/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756612.695930:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756612.695936:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756612.695942:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654759680:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756612.695953:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654759808:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787756612.695995:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756612.696019:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756612/real 1787756612] req@ffff8c05a9f74a80 x1874588654759808/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756612.696043:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588654759808/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756612.696059:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756612.696065:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756612.696071:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654759808:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787756612.696083:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654759936:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787756612.696123:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756612.696175:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756612/real 1787756612] req@ffff8c05a9f76680 x1874588654759936/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756612.696200:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588654759936/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756612.696217:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756612.696223:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756612.696230:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654759936:192.168.201.107@tcp:8:kworker.0 00010000:00010000:2.0:1787756614.294193:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787756617.814213:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654760064:192.168.201.107@tcp:250:kworker.0 00000800:00000100:3.0:1787756617.814287:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756617.814367:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756617/real 1787756617] req@ffff8c05a9f77100 x1874588654760064/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756617.814407:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588654760064/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756617.814428:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756617.814436:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756617.814441:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654760064:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787756617.814468:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654760192:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787756617.814500:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756617.814532:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756617/real 1787756617] req@ffff8c05a9f75180 x1874588654760192/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756617.814569:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588654760192/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756617.814595:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756617.814606:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756617.814620:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654760192:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756617.814635:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654760320:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787756617.814673:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756617.814689:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756617/real 1787756617] req@ffff8c05a9f77800 x1874588654760320/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756617.814705:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588654760320/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756617.814716:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756617.814720:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756617.814724:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654760320:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756617.814733:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654760448:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787756617.814760:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756617.814773:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756617/real 1787756617] req@ffff8c05a9f76300 x1874588654760448/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756617.814783:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588654760448/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756617.814790:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756617.814793:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756617.814795:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654760448:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787756617.814799:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654760576:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787756617.814815:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756617.814822:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756617/real 1787756617] req@ffff8c05a9f74e00 x1874588654760576/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756617.814831:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588654760576/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756617.814836:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756617.814838:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756617.814841:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654760576:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756617.814845:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654760704:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787756617.814856:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756617.814863:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756617/real 1787756617] req@ffff8c05a9f75500 x1874588654760704/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756617.814871:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588654760704/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756617.814877:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756617.814879:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756617.814881:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654760704:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756617.814919:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654760832:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787756617.814965:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756617.814975:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756617/real 1787756617] req@ffff8c05a9f75880 x1874588654760832/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756617.814985:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588654760832/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756617.814992:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756617.814995:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756617.814998:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654760832:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787756617.815002:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654760960:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787756617.815016:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756617.815023:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756617/real 1787756617] req@ffff8c05a9f74000 x1874588654760960/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756617.815063:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588654760960/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756617.815093:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756617.815095:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756617.815097:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654760960:192.168.201.107@tcp:8:kworker.0 00010000:00010000:2.0:1787756618.390164:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756620.438165:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756622.486155:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756622.934073:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756622.934086:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787756622.934092:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756622.934104:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9679 00000100:00080000:2.0:1787756622.934118:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756622.934126:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756622.934176:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588654761088/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756622.934216:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756622.934220:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756622.934223:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756622.934228:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9679 00000100:00080000:2.0:1787756622.934232:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756622.934237:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756622.934248:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756622.934264:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588654761216/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756622.934280:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756622.934284:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756622.934287:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756622.934292:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9679 00000100:00080000:2.0:1787756622.934298:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756622.934308:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787756622.934308:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654761088:192.168.201.107@tcp:250:kworker.0 00000008:00100000:2.0:1787756622.934322:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756622.934340:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588654761344/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756622.934354:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756622.934359:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756622.934361:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756622.934366:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9679 00000100:00080000:2.0:1787756622.934385:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756622.934391:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756622.934397:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787756622.934406:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756622.934414:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588654761472/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756622.934430:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756622.934436:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756622.934438:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756622.934442:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9679 00000100:00080000:2.0:1787756622.934446:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756622.934452:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756622.934458:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756622.934471:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588654761600/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787756622.934494:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756622/real 1787756622] req@ffff8c05b2e70000 x1874588654761088/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756622.934503:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756622.934516:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756622.934524:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756622.934539:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9679 00000100:00080000:2.0:1787756622.934546:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756622.934551:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756622.934558:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756622.934563:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588654761088/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756622.934579:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588654761728/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756622.934603:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756622.934604:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756622.934618:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787756622.934624:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787756622.934625:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787756622.934642:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654761088:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787756622.934643:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9679 00000100:00080000:2.0:1787756622.934654:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787756622.934667:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654761216:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787756622.934673:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756622.934682:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756622.934701:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588654761856/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756622.934715:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787756622.934722:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787756622.934724:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756622.934730:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756622.934738:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9679 00000100:00080000:2.0:1787756622.934746:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787756622.934754:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756622/real 1787756622] req@ffff8c05b2e73100 x1874588654761216/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756622.934755:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756622.934767:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756622.934788:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588654761984/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787756622.934801:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588654761216/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756622.934823:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756622.934832:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756622.934842:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654761216:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756622.934933:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654761344:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787756622.935007:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756622.935026:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756622/real 1787756622] req@ffff8c05b2e72d80 x1874588654761344/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756622.935043:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588654761344/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756622.935053:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756622.935057:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756622.935062:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654761344:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756622.935073:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654761472:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787756622.935109:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756622.935133:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756622/real 1787756622] req@ffff8c05b2e72680 x1874588654761472/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756622.935158:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588654761472/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756622.935176:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756622.935182:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756622.935189:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654761472:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787756622.935199:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654761600:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787756622.935229:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756622.935257:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756622/real 1787756622] req@ffff8c05b2e70e00 x1874588654761600/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756622.935272:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588654761600/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756622.935283:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756622.935287:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756622.935291:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654761600:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756622.935298:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654761728:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787756622.935319:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756622.935331:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756622/real 1787756622] req@ffff8c05b2e72a00 x1874588654761728/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756622.935345:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588654761728/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756622.935354:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756622.935358:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756622.935361:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654761728:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756622.935367:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654761856:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787756622.935385:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756622.935396:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756622/real 1787756622] req@ffff8c05b2e71180 x1874588654761856/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756622.935410:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588654761856/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756622.935420:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756622.935424:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756622.935428:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654761856:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787756622.935434:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654761984:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787756622.935451:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756622.935462:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756622/real 1787756622] req@ffff8c05b2e72300 x1874588654761984/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756622.935474:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588654761984/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756622.935484:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756622.935488:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756622.935491:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654761984:192.168.201.107@tcp:8:kworker.0 00010000:00010000:2.0:1787756626.582020:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756628.054037:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756628.054047:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787756628.054051:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756628.054062:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9684 00000100:00080000:3.0:1787756628.054074:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756628.054084:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787756628.054129:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588654762112/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756628.054176:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756628.054184:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756628.054188:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756628.054195:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9684 00000100:00080000:3.0:1787756628.054202:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756628.054209:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756628.054221:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756628.054245:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588654762240/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756628.054264:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756628.054269:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756628.054271:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756628.054278:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9684 00000100:00080000:3.0:1787756628.054283:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756628.054289:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756628.054296:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756628.054312:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588654762368/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756628.054335:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756628.054340:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756628.054343:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756628.054348:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9684 00000100:00080000:3.0:1787756628.054353:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756628.054359:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756628.054448:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756628.054476:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588654762496/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756628.054503:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756628.054508:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756628.054511:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756628.054522:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9684 00000100:00080000:3.0:1787756628.054531:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756628.054537:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756628.054545:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756628.054578:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588654762624/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756628.054760:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756628.054771:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756628.054778:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756628.054792:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9684 00000100:00080000:3.0:1787756628.054920:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756628.054934:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756628.054950:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756628.054977:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588654762752/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756628.055014:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756628.055019:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787756628.055024:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756628.055032:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9684 00000100:00080000:3.0:1787756628.055038:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756628.055050:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756628.055061:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756628.055083:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588654762880/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756628.055105:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756628.055110:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756628.055113:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756628.055122:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9684 00000100:00080000:3.0:1787756628.055131:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756628.055142:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756628.055150:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756628.055178:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588654763008/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787756628.630094:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756630.678133:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756633.173986:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756633.174019:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787756633.174026:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756633.174042:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9689 00000100:00080000:3.0:1787756633.174058:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756633.174070:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787756633.174126:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588654763136/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756633.174165:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756633.174185:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756633.174189:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756633.174198:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9689 00000100:00080000:3.0:1787756633.174206:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756633.174214:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756633.174228:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756633.174255:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588654763264/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756633.174275:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756633.174281:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756633.174284:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756633.174290:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9689 00000100:00080000:3.0:1787756633.174296:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756633.174302:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756633.174310:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756633.174328:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588654763392/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756633.174347:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756633.174352:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756633.174355:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756633.174362:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9689 00000100:00080000:3.0:1787756633.174369:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756633.174377:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756633.174385:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756633.174405:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588654763520/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756633.174422:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756633.174429:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756633.174433:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756633.174440:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9689 00000100:00080000:3.0:1787756633.174446:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756633.174453:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756633.174461:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756633.174477:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588654763648/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756633.174493:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756633.174509:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756633.174513:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756633.174519:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9689 00000100:00080000:3.0:1787756633.174525:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756633.174548:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756633.174558:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756633.174579:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588654763776/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756633.174613:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756633.174620:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787756633.174624:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756633.174631:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9689 00000100:00080000:3.0:1787756633.174638:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756633.174645:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756633.174653:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756633.174671:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588654763904/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756633.174687:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756633.174694:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756633.174697:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756633.174704:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9689 00000100:00080000:3.0:1787756633.174710:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756633.174718:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756633.174727:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756633.174744:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588654764032/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787756634.774045:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756636.822143:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756638.294040:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756638.294057:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787756638.294064:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756638.294080:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9694 00000100:00080000:2.0:1787756638.294097:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756638.294109:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756638.294163:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588654764160/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756638.294205:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756638.294210:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756638.294212:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756638.294218:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9694 00000100:00080000:2.0:1787756638.294223:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756638.294228:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756638.294239:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756638.294258:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588654764288/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756638.294275:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756638.294279:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756638.294282:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756638.294294:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9694 00000100:00080000:2.0:1787756638.294306:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756638.294313:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756638.294319:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756638.294334:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588654764416/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756638.294348:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756638.294353:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756638.294357:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756638.294362:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9694 00000100:00080000:2.0:1787756638.294366:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756638.294370:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756638.294377:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756638.294395:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588654764544/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756638.294415:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756638.294420:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756638.294423:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756638.294429:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9694 00000100:00080000:2.0:1787756638.294436:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756638.294441:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756638.294447:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756638.294470:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588654764672/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756638.294484:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756638.294489:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756638.294496:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756638.294512:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9694 00000100:00080000:2.0:1787756638.294523:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756638.294529:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756638.294535:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756638.294557:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588654764800/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756638.294576:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756638.294586:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787756638.294592:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756638.294603:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9694 00000100:00080000:2.0:1787756638.294613:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756638.294619:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756638.294625:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756638.294638:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588654764928/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756638.294704:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756638.294710:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756638.294713:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756638.294719:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9694 00000100:00080000:2.0:1787756638.294724:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756638.294731:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756638.294744:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756638.294767:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588654765056/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787756638.870175:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756642.966165:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756643.414032:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756643.414049:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787756643.414067:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756643.414080:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9699 00000100:00080000:3.0:1787756643.414095:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756643.414106:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787756643.414149:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588654765184/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756643.414182:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756643.414186:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756643.414189:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756643.414193:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9699 00000100:00080000:3.0:1787756643.414210:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756643.414218:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756643.414231:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756643.414259:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588654765312/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756643.414283:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756643.414289:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756643.414293:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756643.414299:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9699 00000100:00080000:3.0:1787756643.414306:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756643.414311:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756643.414317:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756643.414335:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588654765440/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756643.414356:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756643.414361:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756643.414364:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756643.414369:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9699 00000100:00080000:3.0:1787756643.414373:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756643.414377:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756643.414383:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756643.414394:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588654765568/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756643.414405:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756643.414409:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756643.414412:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756643.414418:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9699 00000100:00080000:3.0:1787756643.414422:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756643.414433:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756643.414442:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756643.414463:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79c00 x1874588654765696/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756643.414485:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756643.414502:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756643.414510:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756643.414521:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9699 00000100:00080000:3.0:1787756643.414533:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756643.414550:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756643.414563:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756643.414615:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588654765824/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756643.414638:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756643.414644:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787756643.414648:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756643.414655:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9699 00000100:00080000:3.0:1787756643.414663:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756643.414677:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756643.414689:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756643.414715:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588654765952/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756643.414747:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756643.414762:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756643.414772:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756643.414787:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9699 00000100:00080000:3.0:1787756643.414799:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756643.414815:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756643.414828:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756643.414860:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588654766080/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787756645.014118:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756647.062138:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756648.534018:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756648.534035:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787756648.534041:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756648.534059:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9704 00000100:00080000:3.0:1787756648.534076:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756648.534088:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787756648.534156:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588654766208/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756648.534200:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756648.534206:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756648.534208:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756648.534214:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9704 00000100:00080000:3.0:1787756648.534218:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756648.534223:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756648.534239:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756648.534261:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588654766336/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756648.534279:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756648.534285:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756648.534293:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756648.534312:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9704 00000100:00080000:3.0:1787756648.534318:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756648.534326:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756648.534337:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756648.534363:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588654766464/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756648.534387:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756648.534394:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756648.534398:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756648.534407:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9704 00000100:00080000:3.0:1787756648.534416:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756648.534430:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756648.534447:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756648.534471:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588654766592/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756648.534485:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756648.534490:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756648.534493:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756648.534526:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9704 00000100:00080000:3.0:1787756648.534534:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756648.534541:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756648.534549:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756648.534572:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79c00 x1874588654766720/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756648.534627:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756648.534633:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756648.534636:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756648.534649:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9704 00000100:00080000:3.0:1787756648.534662:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756648.534674:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756648.534683:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756648.534707:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588654766848/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756648.534735:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756648.534740:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787756648.534743:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756648.534749:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9704 00000100:00080000:3.0:1787756648.534753:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756648.534759:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756648.534769:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756648.534793:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588654766976/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756648.534809:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756648.534815:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756648.534818:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756648.534827:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9704 00000100:00080000:3.0:1787756648.534834:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756648.534843:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756648.534852:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756648.534875:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588654767104/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787756651.158124:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756653.654000:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756653.654013:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787756653.654019:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756653.654032:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9710 00000100:00080000:3.0:1787756653.654049:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756653.654062:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787756653.654140:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588654767232/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756653.654184:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756653.654189:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756653.654192:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756653.654198:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9710 00000100:00080000:3.0:1787756653.654204:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756653.654210:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756653.654221:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756653.654242:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588654767360/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756653.654256:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756653.654261:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756653.654263:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756653.654269:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9710 00000100:00080000:3.0:1787756653.654275:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756653.654280:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756653.654287:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756653.654300:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588654767488/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756653.654319:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756653.654332:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756653.654335:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756653.654341:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9710 00000100:00080000:3.0:1787756653.654345:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756653.654349:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756653.654355:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756653.654372:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588654767616/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756653.654385:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756653.654391:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756653.654393:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756653.654399:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9710 00000100:00080000:3.0:1787756653.654429:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756653.654436:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756653.654443:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756653.654462:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588654767744/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756653.654484:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756653.654489:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756653.654493:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756653.654502:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9710 00000100:00080000:3.0:1787756653.654509:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756653.654515:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756653.654522:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756653.654536:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588654767872/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756653.654549:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756653.654553:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787756653.654555:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756653.654559:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9710 00000100:00080000:3.0:1787756653.654563:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756653.654567:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756653.654574:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756653.654593:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588654768000/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756653.654654:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756653.654659:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756653.654662:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756653.654669:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9710 00000100:00080000:3.0:1787756653.654675:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756653.654683:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756653.654693:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756653.654716:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588654768128/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787756655.254095:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756658.774097:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756658.774112:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787756658.774118:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756658.774131:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9715 00000100:00080000:2.0:1787756658.774144:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756658.774154:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756658.774201:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588654768256/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756658.774260:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756658.774266:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756658.774269:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756658.774275:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9715 00000100:00080000:2.0:1787756658.774279:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756658.774284:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756658.774296:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756658.774343:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588654768384/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756658.774365:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756658.774372:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756658.774377:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756658.774386:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9715 00000100:00080000:2.0:1787756658.774394:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756658.774404:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756658.774417:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756658.774448:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588654768512/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756658.774467:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756658.774472:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756658.774475:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756658.774481:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9715 00000100:00080000:2.0:1787756658.774487:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756658.774492:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756658.774501:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756658.774523:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588654768640/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756658.774540:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756658.774544:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756658.774546:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756658.774550:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9715 00000100:00080000:2.0:1787756658.774556:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756658.774561:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756658.774567:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756658.774583:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588654768768/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756658.774595:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756658.774599:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756658.774602:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756658.774613:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9715 00000100:00080000:2.0:1787756658.774623:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756658.774635:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756658.774648:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756658.774667:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588654768896/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756658.774725:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756658.774739:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787756658.774742:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756658.774754:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9715 00000100:00080000:2.0:1787756658.774762:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756658.774767:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756658.774773:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756658.774806:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588654769024/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756658.774824:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756658.774840:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756658.774844:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756658.774860:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9715 00000100:00080000:2.0:1787756658.774872:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756658.774922:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756658.774929:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756658.774944:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588654769152/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787756659.350164:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756663.446120:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756663.894074:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756663.894088:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787756663.894094:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756663.894108:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9720 00000100:00080000:3.0:1787756663.894123:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756663.894138:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787756663.894205:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588654769280/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756663.894284:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756663.894291:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756663.894294:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756663.894302:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9720 00000100:00080000:3.0:1787756663.894310:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756663.894318:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756663.894332:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756663.894363:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588654769408/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756663.894376:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654769280:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787756663.894391:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756663.894399:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756663.894403:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756663.894412:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9720 00000100:00080000:3.0:1787756663.894420:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756663.894428:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756663.894438:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756663.894456:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588654769536/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756663.894472:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787756663.894478:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787756663.894481:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756663.894487:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756663.894495:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9720 00000100:00080000:3.0:1787756663.894503:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756663.894509:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756663.894562:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756663/real 1787756663] req@ffff8c05a9f79500 x1874588654769280/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756663.894590:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588654769280/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756663.894606:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756663.894613:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756663.894619:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654769280:192.168.201.107@tcp:250:kworker.0 00000008:00100000:3.0:1787756663.894640:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756663.894659:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654769408:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756663.894689:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588654769664/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787756663.894708:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787756663.894724:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787756663.894736:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756663/real 1787756663] req@ffff8c05a9f7b100 x1874588654769408/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756663.894736:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756663.894757:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787756663.894779:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588654769408/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756663.894780:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9720 00000100:00080000:3.0:1787756663.894796:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756663.894800:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756663.894809:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:2.0:1787756663.894811:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756663.894826:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654769408:192.168.201.107@tcp:38:kworker.0 00000008:00100000:3.0:1787756663.894828:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756663.894844:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654769536:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756663.894872:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588654769792/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787756663.894937:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787756663.894956:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787756663.894965:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756663/real 1787756663] req@ffff8c05a9f7b800 x1874588654769536/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756663.894966:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756663.894975:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756663.894992:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9720 00000100:00080000:3.0:1787756663.895001:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787756663.895008:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588654769536/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756663.895015:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756663.895039:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787756663.895043:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756663.895053:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756663.895059:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654769536:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756663.895067:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588654769920/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756663.895086:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787756663.895090:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654769664:192.168.201.107@tcp:8:kworker.0 00000100:00080000:3.0:1787756663.895093:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787756663.895101:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756663.895109:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9720 00000100:00080000:3.0:1787756663.895113:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756663.895119:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756663.895126:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787756663.895134:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756663.895150:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588654770048/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756663.895164:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756663/real 1787756663] req@ffff8c05a9f7bb80 x1874588654769664/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756663.895190:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787756663.895199:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588654769664/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756663.895206:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756663.895217:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756663.895222:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756663.895254:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9720 00000100:00080000:2.0:1787756663.895266:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787756663.895270:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787756663.895282:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654769664:192.168.201.107@tcp:8:kworker.0 00000100:00080000:3.0:1787756663.895284:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756663.895300:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756663.895313:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654769792:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756663.895326:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588654770176/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787756663.895370:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756663.895393:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756663/real 1787756663] req@ffff8c05a9f7ad80 x1874588654769792/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756663.895409:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588654769792/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756663.895420:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756663.895425:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756663.895429:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654769792:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756663.895447:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654769920:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756663.895471:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756663.895482:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756663/real 1787756663] req@ffff8c05a9f79f80 x1874588654769920/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756663.895506:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588654769920/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756663.895516:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756663.895519:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756663.895523:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654769920:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756663.895529:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654770048:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756663.895547:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756663.895566:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756663/real 1787756663] req@ffff8c05a9f79180 x1874588654770048/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756663.895582:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588654770048/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756663.895592:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756663.895596:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756663.895600:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654770048:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756663.895609:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654770176:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756663.895631:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756663.895648:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756663/real 1787756663] req@ffff8c05a9f78e00 x1874588654770176/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756663.895661:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588654770176/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756663.895681:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756663.895686:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756663.895692:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654770176:192.168.201.107@tcp:8:kworker.0 00010000:00010000:3.0:1787756667.542155:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787756669.014453:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654770304:192.168.201.107@tcp:250:kworker.0 00000800:00000100:2.0:1787756669.014601:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756669.014690:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756669/real 1787756669] req@ffff8c05b2d9a300 x1874588654770304/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756669.014744:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588654770304/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756669.014785:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756669.014810:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756669.014838:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654770304:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787756669.014973:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654770432:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756669.015052:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756669.015085:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756669/real 1787756669] req@ffff8c05b2d98380 x1874588654770432/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756669.015102:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654770432/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756669.015114:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756669.015119:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756669.015123:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654770432:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756669.015206:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654770560:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756669.015252:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756669.015275:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756669/real 1787756669] req@ffff8c05b2d9b480 x1874588654770560/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756669.015293:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654770560/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756669.015303:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756669.015308:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756669.015312:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654770560:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756669.015321:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654770688:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756669.015348:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756669.015361:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756669/real 1787756669] req@ffff8c05b2d9b100 x1874588654770688/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756669.015375:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654770688/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756669.015386:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756669.015389:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756669.015393:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654770688:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756669.015401:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654770816:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756669.015421:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756669.015433:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756669/real 1787756669] req@ffff8c05b2d9aa00 x1874588654770816/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756669.015449:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588654770816/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756669.015459:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756669.015463:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756669.015466:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654770816:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756669.015473:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654770944:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756669.015493:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756669.015504:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756669/real 1787756669] req@ffff8c05b2d98700 x1874588654770944/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756669.015519:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654770944/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756669.015529:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756669.015532:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756669.015536:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654770944:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756669.015543:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654771072:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756669.015563:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756669.015573:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756669/real 1787756669] req@ffff8c05b2d99f80 x1874588654771072/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756669.015586:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654771072/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756669.015606:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756669.015610:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756669.015615:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654771072:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756669.015621:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654771200:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756669.015640:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756669.015651:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756669/real 1787756669] req@ffff8c05b2d98000 x1874588654771200/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756669.015664:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654771200/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756669.015675:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756669.015679:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756669.015682:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654771200:192.168.201.107@tcp:8:kworker.0 00010000:00010000:2.0:1787756671.638236:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787756674.134307:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654771328:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787756674.134526:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654771456:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756674.134590:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654771584:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756674.134639:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654771712:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756674.134741:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654771840:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756674.134830:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654771968:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756674.134987:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654772096:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756674.135018:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654772224:192.168.201.107@tcp:8:kworker.0 00010000:00010000:3.0:1787756675.734206:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787756677.206653:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756674/real 1787756677] req@ffff8c05b2d9b100 x1874588654772096/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756677.206712:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654772096/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756677.206738:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756677.206749:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756677.206758:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654772096:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756677.206776:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756674/real 1787756677] req@ffff8c05b2d9b480 x1874588654772224/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756677.206802:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654772224/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756677.206818:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756677.206824:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756677.206832:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654772224:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756677.206848:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756674/real 1787756677] req@ffff8c05b2d98700 x1874588654771840/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756677.206977:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654771840/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756677.207048:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756677.207058:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756677.207065:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654771840:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756677.207081:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756674/real 1787756677] req@ffff8c05b2d9aa00 x1874588654771968/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756677.207109:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588654771968/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756677.207124:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756677.207143:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756677.207149:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654771968:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756677.207165:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756674/real 1787756677] req@ffff8c05b2d9bb80 x1874588654771456/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756677.207192:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588654771456/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756677.207209:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756677.207226:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756677.207234:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654771456:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756677.207249:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756674/real 1787756677] req@ffff8c05b2d98000 x1874588654771584/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756677.207278:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654771584/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756677.207297:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756677.207305:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756677.207313:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654771584:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756677.207327:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756674/real 1787756677] req@ffff8c05b2d99f80 x1874588654771712/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756677.207350:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654771712/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756677.207366:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756677.207374:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756677.207380:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654771712:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756677.207395:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756674/real 1787756677] req@ffff8c05b2d99880 x1874588654771328/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756677.207418:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654771328/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756677.207432:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756677.207438:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756677.207445:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654771328:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787756679.254148:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756679.254165:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787756679.254173:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756679.254190:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9735 00000100:00080000:3.0:1787756679.254207:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756679.254219:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787756679.254281:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588654772352/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756679.254324:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756679.254330:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756679.254332:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756679.254338:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9735 00000100:00080000:3.0:1787756679.254343:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756679.254347:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756679.254362:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756679.254377:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588654772480/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756679.254399:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756679.254404:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756679.254407:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756679.254412:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9735 00000100:00080000:3.0:1787756679.254416:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756679.254422:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756679.254428:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756679.254440:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588654772608/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756679.254459:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756679.254466:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756679.254474:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787756679.254477:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654772352:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787756679.254488:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9735 00000100:00080000:3.0:1787756679.254501:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756679.254509:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756679.254520:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756679.254561:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588654772736/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756679.254583:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756679.254590:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756679.254595:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756679.254603:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9735 00000100:00080000:3.0:1787756679.254614:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756679.254624:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:2.0:1787756679.254636:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:3.0:1787756679.254640:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756679.254673:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79c00 x1874588654772864/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756679.254693:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756679.254700:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756679.254704:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756679.254713:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9735 00000100:00080000:3.0:1787756679.254721:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756679.254736:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756679.254738:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756679/real 1787756679] req@ffff8c05a9f78700 x1874588654772352/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787756679.254749:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756679.254788:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588654772992/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756679.254802:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588654772352/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756679.254825:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756679.254838:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787756679.254841:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756679.254843:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756679.254864:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787756679.254866:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9735 00000100:00080000:3.0:1787756679.254934:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756679.254978:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756679.254989:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756679.255010:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654772352:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787756679.255019:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588654773120/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756679.255039:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654772480:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787756679.255075:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756679.255083:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756679.255088:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756679.255100:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9735 00000100:00080000:3.0:1787756679.255111:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787756679.255114:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787756679.255126:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756679.255140:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756679.255155:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756679/real 1787756679] req@ffff8c05a9f7a680 x1874588654772480/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756679.255190:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588654772480/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756679.255206:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756679.255212:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756679.255219:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654772480:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756679.255270:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654772608:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756679.255316:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756679.255329:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588654773248/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756679.255348:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756679/real 1787756679] req@ffff8c05a9f78a80 x1874588654772608/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756679.255391:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588654772608/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756679.255421:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756679.255429:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756679.255437:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654772608:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756679.255450:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654772736:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756679.255495:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756679.255520:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756679/real 1787756679] req@ffff8c05a9f7b480 x1874588654772736/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756679.255547:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588654772736/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756679.255565:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756679.255572:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756679.255579:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654772736:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756679.255592:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654772864:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756679.255634:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756679.255660:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756679/real 1787756679] req@ffff8c05a9f79c00 x1874588654772864/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756679.255686:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79c00 x1874588654772864/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756679.255704:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756679.255710:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756679.255718:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654772864:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756679.255731:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654772992:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756679.255773:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756679.255797:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756679/real 1787756679] req@ffff8c05a9f7aa00 x1874588654772992/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756679.255821:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588654772992/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756679.255838:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756679.255845:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756679.255852:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654772992:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756679.255866:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654773120:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756679.255974:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756679.256002:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756679/real 1787756679] req@ffff8c05a9f79880 x1874588654773120/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756679.256027:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588654773120/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756679.256045:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756679.256051:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756679.256058:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654773120:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756679.256102:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654773248:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756679.256148:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756679.256174:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756679/real 1787756679] req@ffff8c05a9f78380 x1874588654773248/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756679.256199:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588654773248/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756679.256215:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756679.256222:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756679.256230:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654773248:192.168.201.107@tcp:8:kworker.0 00010000:00010000:2.0:1787756679.830203:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756683.926214:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756684.373997:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756684.374008:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787756684.374013:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756684.374024:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9740 00000100:00080000:3.0:1787756684.374036:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756684.374043:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787756684.374085:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588654773376/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756684.374130:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756684.374134:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756684.374136:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756684.374141:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9740 00000100:00080000:3.0:1787756684.374144:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756684.374149:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756684.374159:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756684.374173:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588654773504/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756684.374183:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756684.374198:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756684.374199:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756684.374203:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9740 00000100:00080000:3.0:1787756684.374205:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756684.374208:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756684.374212:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756684.374221:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588654773632/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756684.374231:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756684.374235:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756684.374237:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756684.374240:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9740 00000100:00080000:3.0:1787756684.374244:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756684.374248:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756684.374254:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756684.374284:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588654773760/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756684.374295:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756684.374301:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756684.374303:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756684.374311:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9740 00000100:00080000:3.0:1787756684.374316:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756684.374324:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756684.374335:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654773376:192.168.201.107@tcp:250:kworker.0 00000008:00100000:3.0:1787756684.374337:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756684.374365:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79c00 x1874588654773888/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756684.374392:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756684.374399:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756684.374402:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756684.374411:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9740 00000100:00080000:3.0:1787756684.374418:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756684.374426:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756684.374437:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756684.374463:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588654774016/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787756684.374483:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787756684.374505:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756684.374514:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787756684.374518:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756684.374529:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9740 00000100:00080000:3.0:1787756684.374538:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756684.374550:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756684.374562:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756684.374583:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756684/real 1787756684] req@ffff8c05a9f7a300 x1874588654773376/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756684.374596:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588654774144/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756684.374636:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787756684.374641:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588654773376/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756684.374644:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756684.374657:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756684.374686:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9740 00000100:00080000:2.0:1787756684.374693:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756684.374696:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756684.374706:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787756684.374715:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756684.374723:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654773376:192.168.201.107@tcp:250:kworker.0 00000008:00100000:3.0:1787756684.374735:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756684.374743:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654773504:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756684.374759:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588654774272/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787756684.374780:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756684.374824:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756684/real 1787756684] req@ffff8c05a9f78380 x1874588654773504/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756684.374841:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588654773504/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756684.374853:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756684.374858:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756684.374973:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654773504:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756684.374984:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654773632:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756684.375015:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756684.375031:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756684/real 1787756684] req@ffff8c05a9f79880 x1874588654773632/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756684.375046:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588654773632/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756684.375056:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756684.375060:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756684.375064:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654773632:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756684.375070:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654773760:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756684.375091:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756684.375101:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756684/real 1787756684] req@ffff8c05a9f7aa00 x1874588654773760/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756684.375114:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588654773760/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756684.375142:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756684.375146:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756684.375151:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654773760:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756684.375195:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654773888:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756684.375236:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756684.375249:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756684/real 1787756684] req@ffff8c05a9f79c00 x1874588654773888/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756684.375262:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79c00 x1874588654773888/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756684.375272:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756684.375276:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756684.375280:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654773888:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756684.375303:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654774016:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756684.375325:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756684.375337:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756684/real 1787756684] req@ffff8c05a9f7b480 x1874588654774016/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756684.375351:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588654774016/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756684.375361:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756684.375364:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756684.375368:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654774016:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756684.375374:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654774144:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756684.375393:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756684.375406:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756684/real 1787756684] req@ffff8c05a9f78a80 x1874588654774144/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756684.375420:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588654774144/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756684.375430:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756684.375434:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756684.375437:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654774144:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756684.375444:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654774272:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756684.375462:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756684.375473:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756684/real 1787756684] req@ffff8c05a9f7a680 x1874588654774272/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756684.375487:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588654774272/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756684.375497:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756684.375500:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756684.375504:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654774272:192.168.201.107@tcp:8:kworker.0 00010000:00010000:2.0:1787756688.022119:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756689.494050:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756689.494062:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787756689.494067:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756689.494079:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9745 00000100:00080000:3.0:1787756689.494092:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756689.494100:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787756689.494155:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588654774400/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756689.494190:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756689.494195:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756689.494197:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756689.494202:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9745 00000100:00080000:3.0:1787756689.494207:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756689.494211:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756689.494222:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756689.494238:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588654774528/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756689.494250:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756689.494254:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756689.494266:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756689.494272:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9745 00000100:00080000:3.0:1787756689.494278:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756689.494283:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756689.494289:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756689.494304:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588654774656/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756689.494308:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654774400:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787756689.494350:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756689.494356:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756689.494358:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756689.494374:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9745 00000100:00080000:3.0:1787756689.494380:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756689.494385:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756689.494392:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756689.494408:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588654774784/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756689.494430:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756689.494434:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756689.494437:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756689.494441:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9745 00000100:00080000:3.0:1787756689.494446:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787756689.494454:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787756689.494476:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756689.494484:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756689.494500:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588654774912/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756689.494521:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756689.494525:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756689.494528:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756689.494533:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9745 00000100:00080000:3.0:1787756689.494538:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787756689.494547:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756689/real 1787756689] req@ffff8c05a9f78700 x1874588654774400/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756689.494547:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756689.494560:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756689.494581:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588654775040/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756689.494591:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588654774400/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756689.494600:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756689.494614:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787756689.494621:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756689.494628:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756689.494631:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9745 00000100:00080000:2.0:1787756689.494648:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787756689.494651:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787756689.494661:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654774400:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787756689.494663:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756689.494678:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756689.494684:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654774528:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756689.494696:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588654775168/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756689.494717:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756689.494720:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756689.494722:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756689.494728:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9745 00000100:00080000:3.0:1787756689.494736:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756689.494760:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:2.0:1787756689.494767:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:3.0:1787756689.494803:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756689.494838:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756689/real 1787756689] req@ffff8c05a9f78e00 x1874588654774528/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756689.494849:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588654775296/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756689.494873:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588654774528/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756689.494886:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756689.494932:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756689.494940:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654774528:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756689.494978:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654774656:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756689.495007:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756689.495020:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756689/real 1787756689] req@ffff8c05a9f79180 x1874588654774656/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756689.495034:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588654774656/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756689.495044:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756689.495047:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756689.495051:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654774656:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756689.495059:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654774784:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756689.495077:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756689.495088:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756689/real 1787756689] req@ffff8c05a9f79f80 x1874588654774784/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756689.495101:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588654774784/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756689.495110:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756689.495113:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756689.495117:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654774784:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756689.495123:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654774912:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756689.495140:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756689.495150:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756689/real 1787756689] req@ffff8c05a9f7ad80 x1874588654774912/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756689.495163:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588654774912/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756689.495172:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756689.495175:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756689.495178:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654774912:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756689.495183:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654775040:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756689.495200:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756689.495209:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756689/real 1787756689] req@ffff8c05a9f7bb80 x1874588654775040/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756689.495222:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588654775040/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756689.495241:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756689.495243:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756689.495247:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654775040:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756689.495252:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654775168:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756689.495269:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756689.495278:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756689/real 1787756689] req@ffff8c05a9f7b800 x1874588654775168/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756689.495290:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588654775168/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756689.495299:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756689.495303:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756689.495306:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654775168:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756689.495311:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654775296:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756689.495327:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756689.495337:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756689/real 1787756689] req@ffff8c05a9f7b100 x1874588654775296/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756689.495349:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588654775296/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756689.495386:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756689.495390:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756689.495394:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654775296:192.168.201.107@tcp:8:kworker.0 00010000:00010000:3.0:1787756692.118061:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787756694.614340:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654775424:192.168.201.107@tcp:250:kworker.0 00000800:00000100:2.0:1787756694.614469:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756694.614569:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756694/real 1787756694] req@ffff8c05a9f74000 x1874588654775424/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756694.614608:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588654775424/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756694.614646:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756694.614676:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756694.614711:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654775424:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787756694.614732:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654775552:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756694.614782:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756694.614812:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756694/real 1787756694] req@ffff8c05a9f75880 x1874588654775552/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756694.614842:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588654775552/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756694.614863:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756694.614869:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756694.614924:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654775552:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756694.614959:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654775680:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756694.614988:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756694.615003:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756694/real 1787756694] req@ffff8c05a9f75500 x1874588654775680/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756694.615019:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588654775680/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756694.615029:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756694.615034:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756694.615039:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654775680:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756694.615045:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654775808:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756694.615066:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756694.615077:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756694/real 1787756694] req@ffff8c05a9f74e00 x1874588654775808/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756694.615100:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588654775808/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756694.615117:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756694.615124:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756694.615131:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654775808:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756694.615144:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654775936:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756694.615185:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756694.615208:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756694/real 1787756694] req@ffff8c05a9f76300 x1874588654775936/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756694.615225:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588654775936/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756694.615236:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756694.615239:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756694.615243:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654775936:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756694.615250:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654776064:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756694.615274:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756694.615286:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756694/real 1787756694] req@ffff8c05a9f77800 x1874588654776064/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756694.615300:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588654776064/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756694.615310:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756694.615315:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756694.615321:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654776064:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756694.615333:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654776192:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756694.615368:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756694.615391:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756694/real 1787756694] req@ffff8c05a9f75180 x1874588654776192/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756694.615417:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588654776192/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756694.615440:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756694.615444:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756694.615449:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654776192:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756694.615461:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654776320:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756694.615501:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756694.615524:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756694/real 1787756694] req@ffff8c05a9f77100 x1874588654776320/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756694.615550:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588654776320/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756694.615563:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756694.615567:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756694.615571:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654776320:192.168.201.107@tcp:8:kworker.0 00010000:00010000:2.0:1787756696.214204:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756700.310077:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756704.406203:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756708.502188:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756712.598197:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756716.694136:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787756720.214346:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654780544:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787756720.214369:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756720.214374:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756720.214380:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756720.214393:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588654780800/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756720.214405:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756720.214410:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787756720.214413:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756720.214416:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9776 00000100:00080000:0.0:1787756720.214421:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756720.214425:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756720.214431:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787756720.214442:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756720.214447:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588654780928/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756720.214462:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756720.214466:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787756720.214469:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756720.214473:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9776 00000100:00080000:0.0:1787756720.214477:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756720.214480:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756720.214485:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756720.214501:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588654781056/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756720.214507:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756720/real 1787756720] req@ffff8c05a9f76680 x1874588654780544/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756720.214534:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756720.214543:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787756720.214548:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588654780544/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756720.214549:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756720.214566:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9776 00000100:00080000:0.0:1787756720.214580:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756720.214586:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756720.214591:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:2.0:1787756720.214596:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756720.214604:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654780544:192.168.201.107@tcp:250:kworker.0 00000008:00100000:0.0:1787756720.214606:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756720.214633:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588654781184/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756720.214637:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654780672:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787756720.214666:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756720.214674:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1787756720.214677:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787756720.214680:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756720.214690:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9776 00000100:00080000:0.0:1787756720.214697:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787756720.214703:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756720/real 1787756720] req@ffff8c05a9f74a80 x1874588654780672/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756720.214707:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756720.214720:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756720.214730:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588654780672/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756720.214743:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588654781312/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756720.214746:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756720.214759:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787756720.214771:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787756720.214772:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654780672:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787756720.214781:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787756720.214787:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787756720.214791:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654780800:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787756720.214801:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9776 00000100:00080000:0.0:1787756720.214810:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756720.214817:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756720.214826:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787756720.214837:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756720.214852:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588654781440/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756720.214858:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756720/real 1787756720] req@ffff8c05a9f75c00 x1874588654780800/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756720.214926:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588654780800/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756720.214936:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756720.214941:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756720.214952:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654780800:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756720.214971:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654780928:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756720.215027:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756720.215053:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756720/real 1787756720] req@ffff8c05a9f74380 x1874588654780928/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756720.215077:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588654780928/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756720.215110:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756720.215116:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756720.215122:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654780928:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756720.215132:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654781056:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756720.215159:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756720.215172:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756720/real 1787756720] req@ffff8c05a9f76a00 x1874588654781056/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756720.215185:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588654781056/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756720.215194:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756720.215197:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756720.215200:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654781056:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756720.215219:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654781184:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756720.215280:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756720.215294:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756720/real 1787756720] req@ffff8c05a9f76d80 x1874588654781184/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756720.215308:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588654781184/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756720.215317:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756720.215320:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756720.215323:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654781184:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756720.215330:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654781312:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756720.215347:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756720.215356:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756720/real 1787756720] req@ffff8c05a9f77b80 x1874588654781312/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756720.215369:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588654781312/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756720.215377:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756720.215380:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756720.215383:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654781312:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756720.215388:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654781440:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756720.215404:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756720.215413:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756720/real 1787756720] req@ffff8c05a9f76680 x1874588654781440/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756720.215425:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588654781440/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756720.215440:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756720.215444:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756720.215447:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654781440:192.168.201.107@tcp:8:kworker.0 00010000:00010000:2.0:1787756720.790181:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756724.886026:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756725.334062:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756725.334076:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787756725.334083:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756725.334098:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9781 00000100:00080000:2.0:1787756725.334113:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756725.334124:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756725.334213:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73800 x1874588654781568/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756725.334261:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756725.334266:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756725.334269:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756725.334275:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9781 00000100:00080000:2.0:1787756725.334281:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756725.334288:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756725.334303:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756725.334322:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654781568:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787756725.334325:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588654781696/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756725.334360:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756725.334368:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756725.334371:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756725.334380:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9781 00000100:00080000:2.0:1787756725.334386:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756725.334391:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756725.334411:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756725.334436:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588654781824/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787756725.334459:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787756725.334478:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756725.334488:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756725.334492:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756725.334506:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9781 00000100:00080000:2.0:1787756725.334515:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756725.334525:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756725.334535:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787756725.334542:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756725/real 1787756725] req@ffff8c05b2e73800 x1874588654781568/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756725.334569:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73800 x1874588654781568/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756725.334582:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787756725.334590:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588654781952/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756725.334595:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756725.334617:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654781568:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787756725.334645:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756725.334665:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756725.334669:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756725.334679:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9781 00000100:00080000:2.0:1787756725.334688:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787756725.334691:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654781696:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787756725.334702:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756725.334714:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756725.334735:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588654782080/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787756725.334751:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787756725.334755:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756725.334761:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756725.334765:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756725.334771:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9781 00000100:00100000:0.0:1787756725.334778:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756725/real 1787756725] req@ffff8c05b2e70380 x1874588654781696/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756725.334779:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756725.334794:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756725.334813:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756725.334840:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588654782208/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787756725.334849:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588654781696/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756725.334864:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756725.334883:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756725.334889:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756725.334897:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654781696:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787756725.334934:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787756725.334938:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756725.334947:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9781 00000100:00100000:0.0:1787756725.334955:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654781824:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787756725.334956:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756725.334969:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756725.334984:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756725.335003:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588654782336/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756725.335016:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756725.335026:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1787756725.335032:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787756725.335032:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756725.335048:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9781 00000100:00080000:2.0:1787756725.335055:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756725.335063:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756725.335072:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787756725.335078:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756725/real 1787756725] req@ffff8c05b2e71880 x1874588654781824/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756725.335095:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588654782464/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787756725.335105:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588654781824/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756725.335161:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756725.335171:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756725.335182:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654781824:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756725.335197:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654781952:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756725.335226:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756725.335237:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756725/real 1787756725] req@ffff8c05b2e71c00 x1874588654781952/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756725.335247:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588654781952/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756725.335253:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756725.335256:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756725.335259:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654781952:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756725.335276:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654782080:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756725.335291:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756725.335299:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756725/real 1787756725] req@ffff8c05b2e73b80 x1874588654782080/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756725.335315:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588654782080/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756725.335322:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756725.335325:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756725.335327:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654782080:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756725.335332:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654782208:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756725.335344:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756725.335351:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756725/real 1787756725] req@ffff8c05b2e71500 x1874588654782208/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756725.335361:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588654782208/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756725.335367:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756725.335369:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756725.335372:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654782208:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756725.335377:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654782336:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756725.335389:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756725.335397:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756725/real 1787756725] req@ffff8c05b2e73480 x1874588654782336/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756725.335406:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588654782336/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756725.335417:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756725.335420:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756725.335423:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654782336:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756725.335427:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654782464:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756725.335448:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756725.335462:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756725/real 1787756725] req@ffff8c05b2e70700 x1874588654782464/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756725.335477:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588654782464/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756725.335489:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756725.335493:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756725.335497:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654782464:192.168.201.107@tcp:8:kworker.0 00010000:00010000:0.0:1787756725.910057:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787756728.982162:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787756730.454416:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654782592:192.168.201.107@tcp:250:kworker.0 00000800:00000100:0.0:1787756730.454517:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756730.454582:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756730/real 1787756730] req@ffff8c05b2d98380 x1874588654782592/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756730.454642:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654782592/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756730.454720:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756730.454728:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756730.454744:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654782592:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787756730.454785:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654782720:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756730.454834:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756730.454867:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756730/real 1787756730] req@ffff8c05b2d9a300 x1874588654782720/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756730.454959:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588654782720/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756730.454998:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756730.455025:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756730.455045:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654782720:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756730.455071:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654782848:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756730.455097:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756730.455112:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756730/real 1787756730] req@ffff8c05b2d99880 x1874588654782848/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756730.455138:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654782848/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756730.455231:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756730.455238:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756730.455243:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654782848:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756730.455251:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654782976:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756730.455285:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756730.455298:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756730/real 1787756730] req@ffff8c05b2d99f80 x1874588654782976/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756730.455307:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654782976/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756730.455313:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756730.455315:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756730.455317:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654782976:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756730.455321:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654783104:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756730.455334:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756730.455341:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756730/real 1787756730] req@ffff8c05b2d98000 x1874588654783104/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756730.455348:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654783104/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756730.455354:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756730.455356:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756730.455358:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654783104:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756730.455372:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654783232:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756730.455384:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756730.455390:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756730/real 1787756730] req@ffff8c05b2d9bb80 x1874588654783232/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756730.455397:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588654783232/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756730.455403:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756730.455405:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756730.455407:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654783232:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756730.455411:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654783360:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756730.455421:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756730.455431:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756730/real 1787756730] req@ffff8c05b2d9aa00 x1874588654783360/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756730.455442:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588654783360/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756730.455451:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756730.455455:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756730.455458:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654783360:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756730.455466:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654783488:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756730.455486:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756730.455495:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756730/real 1787756730] req@ffff8c05b2d98700 x1874588654783488/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756730.455503:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654783488/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756730.455509:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756730.455511:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756730.455513:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654783488:192.168.201.107@tcp:8:kworker.0 00010000:00010000:3.0:1787756733.078151:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787756734.102348:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787756735.574334:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654783616:192.168.201.107@tcp:250:kworker.0 00000800:00000100:0.0:1787756735.574488:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756735.574573:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756735/real 1787756735] req@ffff8c05b2d9b480 x1874588654783616/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756735.574664:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654783616/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756735.574747:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756735.574759:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756735.574788:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654783616:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787756735.574849:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654783744:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756735.575035:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756735.575085:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756735/real 1787756735] req@ffff8c05b2d9b100 x1874588654783744/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756735.575113:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654783744/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756735.575123:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756735.575127:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756735.575131:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654783744:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756735.575142:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654783872:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756735.575192:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756735.575213:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756735/real 1787756735] req@ffff8c05b2d98700 x1874588654783872/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756735.575228:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654783872/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756735.575236:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756735.575242:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756735.575247:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654783872:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756735.575253:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654784000:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756735.575291:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756735.575326:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756735/real 1787756735] req@ffff8c05b2d9aa00 x1874588654784000/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756735.575359:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588654784000/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756735.575381:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756735.575387:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756735.575393:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654784000:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756735.575406:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654784128:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756735.575447:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756735.575467:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756735/real 1787756735] req@ffff8c05b2d9bb80 x1874588654784128/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756735.575489:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588654784128/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756735.575499:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756735.575522:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756735.575526:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654784128:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756735.575535:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654784256:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756735.575561:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756735.575579:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756735/real 1787756735] req@ffff8c05b2d98000 x1874588654784256/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756735.575601:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654784256/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756735.575617:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756735.575630:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756735.575637:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654784256:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756735.575693:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654784384:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756735.575728:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756735.575751:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756735/real 1787756735] req@ffff8c05b2d99f80 x1874588654784384/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756735.575780:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654784384/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756735.575797:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756735.575800:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756735.575803:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654784384:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756735.575809:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654784512:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756735.575833:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756735.575848:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756735/real 1787756735] req@ffff8c05b2d99880 x1874588654784512/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756735.575862:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654784512/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756735.575918:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756735.575924:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756735.575929:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654784512:192.168.201.107@tcp:8:kworker.0 00010000:00010000:2.0:1787756737.174167:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756739.222274:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787756740.694467:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654784640:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787756740.694676:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654784768:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756740.694737:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654784896:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756740.694935:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654785024:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756740.695145:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654785152:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756740.695272:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654785280:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756740.695378:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654785408:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756740.695549:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654785536:192.168.201.107@tcp:8:kworker.0 00010000:00010000:3.0:1787756741.270104:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787756742.294401:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787756743.766173:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.107:988 00000400:00000100:2.0:1787756743.766215:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.107@tcp at host 192.168.201.107 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787756743.766232:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787756743.766270:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787756743.766278:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787756743.766286:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787756743.766294:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787756743.766310:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787756743.766321:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787756743.766335:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:0.0:1787756743.766358:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756740/real 1787756743] req@ffff8c05b2d9aa00 x1874588654785408/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756743.766400:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588654785408/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756743.766416:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756743.766425:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756743.766432:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654785408:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756743.766442:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756740/real 1787756743] req@ffff8c05b2d9bb80 x1874588654785280/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756743.766456:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588654785280/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756743.766465:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756743.766469:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756743.766473:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654785280:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756743.766484:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756740/real 1787756743] req@ffff8c05b2d98000 x1874588654785152/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756743.766502:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654785152/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756743.766513:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756743.766517:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756743.766521:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654785152:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756743.766529:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756740/real 1787756743] req@ffff8c05b2d99f80 x1874588654785024/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756743.766542:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654785024/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756743.766557:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756743.766561:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756743.766565:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654785024:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756743.766573:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756740/real 1787756743] req@ffff8c05b2d99880 x1874588654784896/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756743.766586:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654784896/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756743.766593:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756743.766597:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756743.766601:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654784896:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756743.766608:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756740/real 1787756743] req@ffff8c05b2d98380 x1874588654784768/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756743.766620:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654784768/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756743.766627:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756743.766631:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756743.766634:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654784768:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756743.766642:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756740/real 1787756743] req@ffff8c05b2d9a300 x1874588654784640/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756743.766657:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588654784640/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756743.766666:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756743.766670:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756743.766674:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654784640:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787756743.766721:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756740/real 1787756743] req@ffff8c05b2d98700 x1874588654785536/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756743.766740:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654785536/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756743.766749:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756743.766753:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756743.766756:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654785536:192.168.201.107@tcp:8:kworker.0 00000100:00080000:0.0:1787756745.814046:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756745.814059:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787756745.814064:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756745.814077:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9802 00000100:00080000:0.0:1787756745.814113:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756745.814131:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787756745.814188:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588654785664/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756745.814259:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756745.814266:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787756745.814270:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756745.814278:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9802 00000100:00080000:0.0:1787756745.814301:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756745.814307:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756745.814321:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756745.814343:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588654785792/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756745.814359:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756745.814364:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787756745.814367:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756745.814373:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9802 00000100:00080000:0.0:1787756745.814378:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756745.814383:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756745.814390:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756745.814405:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588654785920/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756745.814421:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756745.814427:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787756745.814430:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756745.814436:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9802 00000100:00080000:0.0:1787756745.814458:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756745.814466:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756745.814475:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787756745.814495:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588654786048/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756745.814514:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756745.814535:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787756745.814539:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756745.814547:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9802 00000100:00080000:0.0:1787756745.814556:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756745.814564:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756745.814575:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756745.814580:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654785664:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787756745.814605:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588654786176/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756745.814629:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756745.814637:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787756745.814641:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756745.814649:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9802 00000100:00080000:0.0:1787756745.814659:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756745.814668:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756745.814681:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787756745.814697:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756745.814711:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588654786304/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756745.814734:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756745.814741:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787756745.814745:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756745.814754:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9802 00000100:00080000:0.0:1787756745.814761:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756745.814771:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756745.814838:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756745.814840:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756745/real 1787756745] req@ffff8c05a9f77100 x1874588654785664/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756745.814934:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588654786432/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756745.814962:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756745.814972:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787756745.814974:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588654785664/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756745.814978:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756745.815010:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9802 00000100:00080000:0.0:1787756745.815025:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756745.815040:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756745.815050:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:2.0:1787756745.815059:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000008:00100000:0.0:1787756745.815068:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756745.815075:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654785664:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787756745.815099:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654785792:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756745.815103:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588654786560/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787756745.815183:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756745.815217:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756745/real 1787756745] req@ffff8c05a9f75180 x1874588654785792/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756745.815242:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588654785792/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756745.815269:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756745.815276:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756745.815282:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654785792:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756745.815294:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654785920:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756745.815335:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756745.815356:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756745/real 1787756745] req@ffff8c05a9f77800 x1874588654785920/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756745.815377:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588654785920/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756745.815391:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756745.815397:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756745.815403:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654785920:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756745.815415:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654786048:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756745.815453:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756745.815476:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756745/real 1787756745] req@ffff8c05a9f76300 x1874588654786048/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756745.815499:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588654786048/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756745.815515:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756745.815522:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756745.815528:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654786048:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756745.815573:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654786176:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756745.815614:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756745.815637:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756745/real 1787756745] req@ffff8c05a9f74e00 x1874588654786176/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756745.815659:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588654786176/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756745.815675:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756745.815682:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756745.815687:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654786176:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756745.815698:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654786304:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756745.815737:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756745.815759:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756745/real 1787756745] req@ffff8c05a9f75500 x1874588654786304/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756745.815809:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588654786304/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756745.815826:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756745.815832:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756745.815838:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654786304:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756745.815864:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654786432:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756745.815948:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756745.815975:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756745/real 1787756745] req@ffff8c05a9f75880 x1874588654786432/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756745.816002:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588654786432/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756745.816018:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756745.816024:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756745.816030:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654786432:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756745.816043:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654786560:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756745.816089:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756745.816112:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756745/real 1787756745] req@ffff8c05a9f74000 x1874588654786560/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756745.816135:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588654786560/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756745.816160:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756745.816167:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756745.816172:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654786560:192.168.201.107@tcp:8:kworker.0 00010000:00010000:3.0:1787756747.414002:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756749.462156:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787756750.486265:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756750.934067:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756750.934085:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787756750.934094:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756750.934115:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9807 00000100:00080000:2.0:1787756750.934133:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756750.934144:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756750.934209:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588654786688/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756750.934259:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756750.934264:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756750.934267:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756750.934273:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9807 00000100:00080000:2.0:1787756750.934278:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756750.934284:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756750.934299:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756750.934317:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588654786816/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787756750.934326:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654786688:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787756750.934339:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756750.934346:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756750.934350:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756750.934357:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9807 00000100:00080000:2.0:1787756750.934362:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756750.934367:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756750.934373:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756750.934395:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588654786944/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756750.934414:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756750.934420:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756750.934425:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756750.934433:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9807 00000100:00080000:2.0:1787756750.934441:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787756750.934450:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787756750.934454:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756750.934469:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756750.934496:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588654787072/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756750.934516:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756750.934522:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756750.934526:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756750.934534:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9807 00000100:00080000:2.0:1787756750.934542:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756750.934550:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756750.934560:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756750.934587:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756750/real 1787756750] req@ffff8c05b2e70000 x1874588654786688/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756750.934591:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588654787200/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756750.934657:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756750.934666:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756750.934671:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787756750.934679:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588654786688/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756750.934724:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9807 00000100:00080000:2.0:1787756750.934740:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756750.934758:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756750.934758:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787756750.934779:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756750.934792:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654786688:192.168.201.107@tcp:250:kworker.0 00000008:00100000:2.0:1787756750.934808:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756750.934844:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588654787328/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787756750.934852:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654786816:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756750.934969:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787756750.934977:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756750.934987:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787756750.934993:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756750.935010:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9807 00000100:00100000:0.0:1787756750.935011:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756750/real 1787756750] req@ffff8c05b2e73100 x1874588654786816/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756750.935023:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756750.935043:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756750.935056:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787756750.935057:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588654786816/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756750.935086:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787756750.935093:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588654787456/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756750.935101:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756750.935118:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654786816:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787756750.935129:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756750.935138:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787756750.935143:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654786944:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787756750.935146:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756750.935167:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9807 00000100:00080000:2.0:1787756750.935177:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756750.935187:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756750.935199:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787756750.935211:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756750.935233:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588654787584/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787756750.935247:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756750/real 1787756750] req@ffff8c05b2e72d80 x1874588654786944/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756750.935299:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588654786944/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756750.935315:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756750.935322:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756750.935329:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654786944:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756750.935342:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654787072:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756750.935386:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756750.935415:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756750/real 1787756750] req@ffff8c05b2e72680 x1874588654787072/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756750.935441:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588654787072/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756750.935456:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756750.935462:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756750.935469:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654787072:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756750.935485:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654787200:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756750.935533:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756750.935557:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756750/real 1787756750] req@ffff8c05b2e70e00 x1874588654787200/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756750.935582:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588654787200/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756750.935613:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756750.935620:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756750.935627:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654787200:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756750.935666:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654787328:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756750.935734:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756750.935758:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756750/real 1787756750] req@ffff8c05b2e72a00 x1874588654787328/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756750.935779:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588654787328/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756750.935795:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756750.935802:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756750.935808:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654787328:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756750.935820:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654787456:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756750.935857:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756750.935926:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756750/real 1787756750] req@ffff8c05b2e71180 x1874588654787456/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756750.935954:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588654787456/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756750.935970:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756750.935977:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756750.935984:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654787456:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756750.935997:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654787584:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756750.936039:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756750.936063:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756750/real 1787756750] req@ffff8c05b2e72300 x1874588654787584/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756750.936087:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588654787584/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756750.936105:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756750.936113:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756750.936120:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654787584:192.168.201.107@tcp:8:kworker.0 00010000:00010000:3.0:1787756755.606161:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756756.054056:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756756.054069:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787756756.054074:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756756.054086:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9812 00000100:00080000:2.0:1787756756.054100:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756756.054110:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756756.054168:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588654787712/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756756.054207:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756756.054212:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756756.054215:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756756.054220:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9812 00000100:00080000:2.0:1787756756.054224:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756756.054228:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756756.054240:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756756.054261:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588654787840/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756756.054276:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756756.054282:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756756.054284:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756756.054293:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9812 00000100:00080000:2.0:1787756756.054304:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787756756.054306:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654787712:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787756756.054313:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756756.054327:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756756.054344:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588654787968/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756756.054360:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756756.054365:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756756.054367:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756756.054373:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9812 00000100:00080000:2.0:1787756756.054380:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756756.054385:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756756.054392:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756756.054407:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588654788096/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756756.054419:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756756.054424:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756756.054427:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756756.054431:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9812 00000100:00080000:2.0:1787756756.054436:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756756.054441:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:0.0:1787756756.054452:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:2.0:1787756756.054459:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756756.054483:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588654788224/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756756.054498:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756756.054503:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756756.054507:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756756.054514:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9812 00000100:00080000:2.0:1787756756.054518:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756756.054522:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756756.054528:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756756.054541:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756756/real 1787756756] req@ffff8c05b2e70a80 x1874588654787712/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756756.054544:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588654788352/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756756.054652:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756756.054658:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787756756.054661:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756756.054667:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9812 00000100:00080000:2.0:1787756756.054672:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756756.054677:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787756756.054686:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588654787712/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787756756.054689:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787756756.054726:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787756756.054735:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588654788480/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756756.054745:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756756.054771:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654787712:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787756756.054785:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756756.054791:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756756.054796:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756756.054802:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9812 00000100:00080000:2.0:1787756756.054809:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756756.054813:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756756.054819:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787756756.054831:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654787840:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756756.054837:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588654788608/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787756756.054954:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756756.054989:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756756/real 1787756756] req@ffff8c05b2e72300 x1874588654787840/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756756.055018:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588654787840/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756756.055030:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756756.055036:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756756.055041:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654787840:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756756.055050:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654787968:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756756.055083:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756756.055097:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756756/real 1787756756] req@ffff8c05b2e71180 x1874588654787968/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756756.055113:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588654787968/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756756.055123:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756756.055127:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756756.055131:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654787968:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756756.055139:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654788096:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756756.055164:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756756.055184:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756756/real 1787756756] req@ffff8c05b2e72a00 x1874588654788096/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756756.055207:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588654788096/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756756.055226:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756756.055234:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756756.055241:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654788096:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756756.055254:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654788224:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756756.055290:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756756.055306:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756756/real 1787756756] req@ffff8c05b2e70e00 x1874588654788224/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756756.055320:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588654788224/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756756.055332:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756756.055336:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756756.055340:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654788224:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756756.055348:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654788352:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756756.055370:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756756.055381:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756756/real 1787756756] req@ffff8c05b2e72680 x1874588654788352/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756756.055396:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588654788352/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756756.055406:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756756.055409:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756756.055414:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654788352:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756756.055422:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654788480:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756756.055442:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756756.055452:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756756/real 1787756756] req@ffff8c05b2e72d80 x1874588654788480/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756756.055466:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588654788480/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756756.055476:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756756.055479:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756756.055483:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654788480:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756756.055513:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654788608:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756756.055533:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756756.055545:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756756/real 1787756756] req@ffff8c05b2e73100 x1874588654788608/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756756.055560:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588654788608/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756756.055570:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756756.055573:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756756.055577:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654788608:192.168.201.107@tcp:8:kworker.0 00010000:00010000:3.0:1787756757.654171:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787756758.678278:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787756761.174137:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756761.174162:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787756761.174171:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756761.174193:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9817 00000100:00080000:0.0:1787756761.174215:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756761.174232:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787756761.174311:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588654788736/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756761.174361:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756761.174369:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787756761.174374:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756761.174386:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9817 00000100:00080000:0.0:1787756761.174397:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756761.174408:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756761.174431:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654788736:192.168.201.107@tcp:250:kworker.0 00000008:00100000:0.0:1787756761.174446:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756761.174482:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588654788864/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756761.174503:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756761.174534:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787756761.174538:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756761.174545:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9817 00000100:00080000:0.0:1787756761.174555:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756761.174562:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756761.174572:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787756761.174591:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756761.174598:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588654788992/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756761.174617:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756761.174623:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787756761.174626:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756761.174632:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9817 00000100:00080000:0.0:1787756761.174638:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756761.174644:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756761.174652:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756761.174671:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756761/real 1787756761] req@ffff8c05a9f74000 x1874588654788736/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756761.174673:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588654789120/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756761.174746:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756761.174758:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1787756761.174758:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588654788736/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756761.174769:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756761.174788:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756761.174791:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9817 00000100:00080000:2.0:1787756761.174805:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787756761.174810:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787756761.174819:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654788736:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787756761.174833:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756761.174848:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756761.174865:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654788864:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756761.174939:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588654789248/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756761.174959:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756761.174965:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787756761.174972:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787756761.174975:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787756761.174985:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9817 00000100:00080000:0.0:1787756761.174998:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756761.175007:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756761.175016:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756761.175026:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756761/real 1787756761] req@ffff8c05a9f75880 x1874588654788864/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756761.175041:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588654789376/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756761.175080:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756761.175087:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787756761.175091:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588654788864/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756761.175092:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756761.175108:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9817 00000100:00080000:0.0:1787756761.175119:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756761.175131:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756761.175147:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787756761.175159:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756761.175172:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756761.175181:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588654789504/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756761.175188:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654788864:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756761.175215:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654788992:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787756761.175225:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756761.175235:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787756761.175240:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756761.175252:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9817 00000100:00080000:0.0:1787756761.175262:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787756761.175269:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787756761.175277:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756761.175290:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756761.175305:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756761/real 1787756761] req@ffff8c05a9f75500 x1874588654788992/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756761.175314:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588654789632/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756761.175365:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588654788992/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756761.175399:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756761.175408:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756761.175418:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654788992:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756761.175434:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654789120:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756761.175476:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756761.175498:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756761/real 1787756761] req@ffff8c05a9f74e00 x1874588654789120/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756761.175555:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588654789120/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756761.175573:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756761.175581:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756761.175590:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654789120:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756761.175623:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654789248:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756761.175665:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756761.175688:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756761/real 1787756761] req@ffff8c05a9f76300 x1874588654789248/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756761.175713:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588654789248/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756761.175731:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756761.175738:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756761.175745:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654789248:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756761.175758:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654789376:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756761.175794:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756761.175816:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756761/real 1787756761] req@ffff8c05a9f77800 x1874588654789376/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756761.175842:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588654789376/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756761.175860:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756761.175867:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756761.175874:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654789376:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756761.175932:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654789504:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756761.175977:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756761.176002:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756761/real 1787756761] req@ffff8c05a9f75180 x1874588654789504/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756761.176029:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588654789504/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756761.176048:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756761.176055:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756761.176063:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654789504:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756761.176076:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654789632:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756761.176113:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756761.176134:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756761/real 1787756761] req@ffff8c05a9f77100 x1874588654789632/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756761.176160:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588654789632/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756761.176178:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756761.176185:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756761.176193:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654789632:192.168.201.107@tcp:8:kworker.0 00010000:00010000:3.0:1787756763.798264:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756765.846117:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756766.294075:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756766.294093:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787756766.294101:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756766.294118:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9822 00000100:00080000:3.0:1787756766.294138:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756766.294153:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787756766.294233:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588654789760/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756766.294282:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756766.294287:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756766.294290:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756766.294297:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9822 00000100:00080000:3.0:1787756766.294302:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756766.294309:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756766.294323:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756766.294365:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588654789888/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756766.294368:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654789760:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787756766.294399:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756766.294405:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756766.294408:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756766.294414:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9822 00000100:00080000:3.0:1787756766.294458:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756766.294464:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756766.294473:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756766.294495:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588654790016/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756766.294511:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756766.294516:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756766.294519:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756766.294526:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9822 00000100:00080000:3.0:1787756766.294531:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756766.294535:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756766.294543:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787756766.294554:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756766.294567:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588654790144/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756766.294582:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756766.294588:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756766.294591:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756766.294595:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9822 00000100:00080000:3.0:1787756766.294599:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756766.294604:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756766.294611:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756766.294624:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588654790272/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756766.294636:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756766.294641:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756766.294644:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756766.294650:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9822 00000100:00080000:3.0:1787756766.294655:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787756766.294660:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756766/real 1787756766] req@ffff8c05a9f79500 x1874588654789760/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756766.294664:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756766.294671:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756766.294701:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588654790400/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756766.294715:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756766.294719:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787756766.294722:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787756766.294727:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588654789760/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756766.294735:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9822 00000100:00080000:3.0:1787756766.294746:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756766.294757:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756766.294765:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787756766.294767:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756766.294777:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756766.294783:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588654790528/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756766.294792:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654789760:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787756766.294817:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756766.294822:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756766.294825:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756766.294831:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9822 00000100:00080000:3.0:1787756766.294834:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756766.294839:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756766.294845:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756766.294847:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654789888:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756766.294869:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588654790656/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787756766.294952:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756766.294986:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756766/real 1787756766] req@ffff8c05a9f7b100 x1874588654789888/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756766.295022:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588654789888/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756766.295034:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756766.295039:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756766.295043:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654789888:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756766.295051:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654790016:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756766.295076:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756766.295089:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756766/real 1787756766] req@ffff8c05a9f7b800 x1874588654790016/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756766.295104:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588654790016/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756766.295114:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756766.295118:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756766.295122:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654790016:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756766.295131:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654790144:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756766.295152:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756766.295163:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756766/real 1787756766] req@ffff8c05a9f7bb80 x1874588654790144/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756766.295177:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588654790144/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756766.295188:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756766.295191:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756766.295194:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654790144:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756766.295201:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654790272:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756766.295221:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756766.295232:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756766/real 1787756766] req@ffff8c05a9f7ad80 x1874588654790272/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756766.295246:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588654790272/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756766.295256:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756766.295259:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756766.295263:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654790272:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756766.295269:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654790400:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756766.295288:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756766.295299:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756766/real 1787756766] req@ffff8c05a9f79f80 x1874588654790400/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756766.295314:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588654790400/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756766.295324:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756766.295328:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756766.295331:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654790400:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756766.295337:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654790528:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756766.295363:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756766.295383:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756766/real 1787756766] req@ffff8c05a9f79180 x1874588654790528/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756766.295407:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588654790528/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756766.295449:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756766.295456:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756766.295463:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654790528:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756766.295502:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654790656:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756766.295533:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756766.295548:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756766/real 1787756766] req@ffff8c05a9f78e00 x1874588654790656/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756766.295563:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588654790656/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756766.295582:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756766.295586:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756766.295590:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654790656:192.168.201.107@tcp:8:kworker.0 00010000:00010000:0.0:1787756766.870286:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756771.414022:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756771.414036:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787756771.414041:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756771.414053:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9827 00000100:00080000:3.0:1787756771.414067:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756771.414075:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787756771.414124:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588654790784/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756771.414160:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756771.414164:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756771.414167:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756771.414172:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9827 00000100:00080000:3.0:1787756771.414176:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756771.414180:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756771.414192:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756771.414211:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588654790912/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756771.414225:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756771.414230:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756771.414233:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756771.414238:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9827 00000100:00080000:3.0:1787756771.414242:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756771.414249:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756771.414256:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756771.414274:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588654791040/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756771.414285:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654790784:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787756771.414297:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756771.414307:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756771.414311:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756771.414321:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9827 00000100:00080000:3.0:1787756771.414353:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756771.414364:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756771.414377:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756771.414414:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588654791168/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756771.414431:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756771.414437:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756771.414440:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756771.414448:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9827 00000100:00080000:3.0:1787756771.414455:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756771.414462:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:2.0:1787756771.414471:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:3.0:1787756771.414480:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756771.414501:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79c00 x1874588654791296/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756771.414515:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756771.414520:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756771.414523:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756771.414531:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9827 00000100:00080000:3.0:1787756771.414540:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756771.414545:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756771.414552:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756771.414562:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756771/real 1787756771] req@ffff8c05a9f78700 x1874588654790784/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756771.414578:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588654791424/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756771.414606:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756771.414610:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787756771.414619:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588654790784/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756771.414620:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756771.414650:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9827 00000100:00080000:3.0:1787756771.414667:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756771.414676:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756771.414677:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:2.0:1787756771.414696:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000008:00100000:3.0:1787756771.414697:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756771.414708:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654790784:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787756771.414727:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654790912:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756771.414743:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588654791552/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756771.414772:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787756771.414773:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787756771.414790:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756771.414796:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787756771.414801:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756771/real 1787756771] req@ffff8c05a9f7a680 x1874588654790912/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756771.414814:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9827 00000100:00100000:2.0:1787756771.414830:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588654790912/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756771.414832:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756771.414856:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:2.0:1787756771.414857:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787756771.414947:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787756771.414969:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756771.414978:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588654791680/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756771.414986:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654790912:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756771.415068:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654791040:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756771.415151:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756771.415193:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756771/real 1787756771] req@ffff8c05a9f78a80 x1874588654791040/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756771.415225:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588654791040/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756771.415245:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756771.415254:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756771.415262:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654791040:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756771.415281:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654791168:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756771.415372:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756771.415412:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756771/real 1787756771] req@ffff8c05a9f7b480 x1874588654791168/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756771.415442:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588654791168/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756771.415464:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756771.415474:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756771.415484:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654791168:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756771.415501:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654791296:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756771.415545:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756771.415566:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756771/real 1787756771] req@ffff8c05a9f79c00 x1874588654791296/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756771.415591:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79c00 x1874588654791296/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756771.415609:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756771.415616:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756771.415622:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654791296:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756771.415634:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654791424:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756771.415664:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756771.415690:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756771/real 1787756771] req@ffff8c05a9f7aa00 x1874588654791424/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756771.415708:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588654791424/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756771.415718:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756771.415724:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756771.415729:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654791424:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756771.415741:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654791552:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756771.415766:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756771.415785:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756771/real 1787756771] req@ffff8c05a9f79880 x1874588654791552/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756771.415800:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588654791552/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756771.415820:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756771.415825:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756771.415828:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654791552:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756771.415836:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654791680:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756771.415860:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756771.415871:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756771/real 1787756771] req@ffff8c05a9f78380 x1874588654791680/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756771.415932:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588654791680/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756771.415951:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756771.415961:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756771.415967:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654791680:192.168.201.107@tcp:8:kworker.0 00010000:00010000:3.0:1787756771.990112:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756774.038132:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787756775.062227:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787756776.534340:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654791808:192.168.201.107@tcp:250:kworker.0 00000800:00000100:2.0:1787756776.534444:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756776.534523:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756776/real 1787756776] req@ffff8c05b2d9b100 x1874588654791808/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756776.534602:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654791808/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756776.534630:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756776.534641:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756776.534654:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654791808:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787756776.534683:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654791936:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756776.534705:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756776.534715:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756776/real 1787756776] req@ffff8c05b2d9b480 x1874588654791936/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756776.534741:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654791936/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756776.534750:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756776.534754:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756776.534758:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654791936:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756776.534785:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654792064:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756776.534804:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756776.534815:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756776/real 1787756776] req@ffff8c05b2d98700 x1874588654792064/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756776.534827:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654792064/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756776.534835:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756776.534839:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756776.534842:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654792064:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756776.534848:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654792192:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756776.534863:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756776.534872:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756776/real 1787756776] req@ffff8c05b2d9a300 x1874588654792192/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756776.534900:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588654792192/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756776.534906:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756776.534908:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756776.534910:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654792192:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756776.534913:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654792320:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756776.534924:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756776.534931:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756776/real 1787756776] req@ffff8c05b2d98380 x1874588654792320/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756776.534939:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654792320/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756776.534944:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756776.534946:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756776.534963:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654792320:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756776.534969:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654792448:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756776.535022:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756776.535035:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756776/real 1787756776] req@ffff8c05b2d99880 x1874588654792448/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756776.535046:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654792448/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756776.535061:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756776.535064:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756776.535067:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654792448:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756776.535072:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654792576:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756776.535089:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756776.535098:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756776/real 1787756776] req@ffff8c05b2d99f80 x1874588654792576/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756776.535108:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654792576/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756776.535116:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756776.535118:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756776.535121:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654792576:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756776.535126:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654792704:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756776.535139:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756776.535147:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756776/real 1787756776] req@ffff8c05b2d98000 x1874588654792704/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756776.535157:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654792704/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756776.535164:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756776.535167:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756776.535170:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654792704:192.168.201.107@tcp:8:kworker.0 00010000:00010000:3.0:1787756780.182192:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787756781.654072:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756781.654089:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787756781.654096:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756781.654112:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9838 00000100:00080000:0.0:1787756781.654130:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756781.654141:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787756781.654231:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588654792832/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756781.654274:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756781.654279:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787756781.654282:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756781.654288:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9838 00000100:00080000:0.0:1787756781.654293:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756781.654299:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756781.654313:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756781.654329:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588654792960/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756781.654344:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756781.654351:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787756781.654353:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756781.654358:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9838 00000100:00080000:0.0:1787756781.654364:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756781.654370:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756781.654376:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756781.654387:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588654793088/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756781.654406:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654792832:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787756781.654411:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756781.654434:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787756781.654438:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756781.654444:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9838 00000100:00080000:0.0:1787756781.654449:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756781.654455:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756781.654462:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787756781.654479:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588654793216/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756781.654493:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756781.654498:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787756781.654500:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756781.654507:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9838 00000100:00080000:0.0:1787756781.654512:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756781.654518:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756781.654524:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787756781.654528:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756781.654548:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588654793344/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756781.654576:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756781.654580:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787756781.654584:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756781.654589:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9838 00000100:00080000:0.0:1787756781.654613:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756781.654623:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756781.654630:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756781/real 1787756781] req@ffff8c05a9f76680 x1874588654792832/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787756781.654639:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756781.654675:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588654792832/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756781.654688:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588654793472/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756781.654698:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756781.654729:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787756781.654737:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787756781.654749:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654792832:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787756781.654752:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787756781.654762:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756781.654775:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9838 00000100:00080000:0.0:1787756781.654785:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787756781.654787:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654792960:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787756781.654803:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756781.654817:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787756781.654849:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588654793600/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787756781.654850:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787756781.654939:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756781.654945:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787756781.654949:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787756781.654962:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756781/real 1787756781] req@ffff8c05a9f77b80 x1874588654792960/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756781.654964:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9838 00000100:00080000:0.0:1787756781.654982:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756781.655006:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756781.655017:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756781.655023:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588654792960/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756781.655042:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787756781.655049:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588654793728/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756781.655057:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756781.655084:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654792960:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756781.655169:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654793088:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756781.655231:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756781.655259:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756781/real 1787756781] req@ffff8c05a9f76d80 x1874588654793088/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756781.655277:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588654793088/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756781.655288:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756781.655293:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756781.655297:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654793088:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756781.655306:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654793216:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756781.655335:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756781.655348:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756781/real 1787756781] req@ffff8c05a9f76a00 x1874588654793216/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756781.655363:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588654793216/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756781.655373:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756781.655377:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756781.655380:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654793216:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756781.655387:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654793344:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756781.655407:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756781.655418:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756781/real 1787756781] req@ffff8c05a9f74380 x1874588654793344/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756781.655432:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588654793344/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756781.655443:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756781.655447:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756781.655450:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654793344:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756781.655457:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654793472:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756781.655475:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756781.655486:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756781/real 1787756781] req@ffff8c05a9f75c00 x1874588654793472/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756781.655499:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588654793472/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756781.655509:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756781.655513:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756781.655517:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654793472:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756781.655524:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654793600:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756781.655543:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756781.655554:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756781/real 1787756781] req@ffff8c05a9f74a80 x1874588654793600/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756781.655568:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588654793600/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756781.655577:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756781.655580:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756781.655585:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654793600:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756781.655593:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654793728:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756781.655611:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756781.655622:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756781/real 1787756781] req@ffff8c05a9f77100 x1874588654793728/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756781.655636:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588654793728/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756781.655646:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756781.655650:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756781.655653:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654793728:192.168.201.107@tcp:8:kworker.0 00010000:00010000:3.0:1787756782.230134:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787756783.254300:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787756786.774081:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756786.774099:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787756786.774106:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756786.774123:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9843 00000100:00080000:0.0:1787756786.774141:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756786.774154:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787756786.774206:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588654793856/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756786.774249:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756786.774257:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787756786.774260:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756786.774269:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9843 00000100:00080000:0.0:1787756786.774277:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756786.774287:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756786.774304:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756786.774332:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588654793984/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756786.774356:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654793856:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787756786.774368:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756786.774375:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787756786.774378:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756786.774385:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9843 00000100:00080000:0.0:1787756786.774391:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756786.774398:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756786.774407:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756786.774426:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588654794112/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756786.774447:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756786.774452:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787756786.774456:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787756786.774462:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787756786.774463:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9843 00000100:00080000:0.0:1787756786.774469:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756786.774476:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756786.774484:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787756786.774502:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588654794240/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756786.774520:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756786.774527:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787756786.774530:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756786.774536:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9843 00000100:00080000:0.0:1787756786.774543:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787756786.774543:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756786/real 1787756786] req@ffff8c05a9f75180 x1874588654793856/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756786.774551:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756786.774561:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756786.774585:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588654794368/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756786.774585:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588654793856/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756786.774606:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756786.774608:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756786.774613:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787756786.774616:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756786.774616:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787756786.774623:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9843 00000100:00100000:2.0:1787756786.774624:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654793856:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787756786.774630:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756786.774635:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756786.774643:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756786.774660:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588654794496/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756786.774660:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654793984:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787756786.774698:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787756786.774703:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787756786.774705:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787756786.774709:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756786.774716:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9843 00000100:00080000:0.0:1787756786.774722:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787756786.774729:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756786/real 1787756786] req@ffff8c05a9f77800 x1874588654793984/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756786.774730:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756786.774740:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756786.774755:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588654793984/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756786.774760:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588654794624/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756786.774774:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756786.774781:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756786.774788:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654793984:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787756786.774795:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787756786.774801:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654794112:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787756786.774802:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787756786.774806:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756786.774813:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9843 00000100:00080000:0.0:1787756786.774819:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756786.774826:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756786.774835:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787756786.774837:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756786.774855:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588654794752/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756786.774859:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756786/real 1787756786] req@ffff8c05a9f76300 x1874588654794112/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756786.774954:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588654794112/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756786.774972:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756786.774980:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756786.774988:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654794112:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756786.775001:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654794240:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756786.775049:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756786.775099:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756786/real 1787756786] req@ffff8c05a9f74e00 x1874588654794240/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756786.775115:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588654794240/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756786.775127:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756786.775131:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756786.775136:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654794240:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756786.775144:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654794368:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756786.775170:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756786.775183:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756786/real 1787756786] req@ffff8c05a9f75500 x1874588654794368/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756786.775197:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588654794368/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756786.775207:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756786.775210:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756786.775214:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654794368:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756786.775237:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654794496:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756786.775258:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756786.775270:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756786/real 1787756786] req@ffff8c05a9f75880 x1874588654794496/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756786.775284:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588654794496/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756786.775294:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756786.775297:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756786.775302:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654794496:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756786.775309:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654794624:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756786.775328:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756786.775339:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756786/real 1787756786] req@ffff8c05a9f74000 x1874588654794624/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756786.775353:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588654794624/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756786.775363:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756786.775366:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756786.775369:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654794624:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756786.775375:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654794752:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756786.775393:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756786.775403:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756786/real 1787756786] req@ffff8c05a9f75180 x1874588654794752/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756786.775417:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588654794752/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756786.775426:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756786.775430:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756786.775433:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654794752:192.168.201.107@tcp:8:kworker.0 00010000:00010000:3.0:1787756788.374158:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756790.422137:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787756791.446298:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787756791.894359:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654794880:192.168.201.107@tcp:250:kworker.0 00000800:00000100:2.0:1787756791.894462:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756791.894525:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756791/real 1787756791] req@ffff8c05b2d9bb80 x1874588654794880/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756791.894564:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588654794880/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756791.894590:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756791.894597:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756791.894609:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654794880:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787756791.894630:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654795008:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756791.894675:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756791.894701:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756791/real 1787756791] req@ffff8c05b2d9aa00 x1874588654795008/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756791.894728:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588654795008/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756791.894756:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756791.894764:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756791.894774:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654795008:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756791.894814:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654795136:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756791.894857:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756791.894995:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756791/real 1787756791] req@ffff8c05b2d98000 x1874588654795136/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756791.895047:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654795136/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756791.895071:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756791.895076:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756791.895081:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654795136:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756791.895091:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654795264:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756791.895125:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756791.895140:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756791/real 1787756791] req@ffff8c05b2d99f80 x1874588654795264/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756791.895154:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654795264/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756791.895164:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756791.895168:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756791.895172:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654795264:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756791.895178:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654795392:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756791.895197:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756791.895209:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756791/real 1787756791] req@ffff8c05b2d99880 x1874588654795392/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756791.895222:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654795392/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756791.895232:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756791.895235:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756791.895240:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654795392:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756791.895260:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654795520:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756791.895280:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756791.895292:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756791/real 1787756791] req@ffff8c05b2d98380 x1874588654795520/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756791.895305:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654795520/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756791.895315:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756791.895318:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756791.895322:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654795520:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756791.895328:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654795648:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756791.895347:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756791.895358:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756791/real 1787756791] req@ffff8c05b2d9a300 x1874588654795648/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756791.895372:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588654795648/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756791.895382:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756791.895385:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756791.895389:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654795648:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756791.895394:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654795776:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756791.895412:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756791.895424:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756791/real 1787756791] req@ffff8c05b2d98700 x1874588654795776/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756791.895437:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654795776/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756791.895447:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756791.895450:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756791.895454:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654795776:192.168.201.107@tcp:8:kworker.0 00010000:00010000:3.0:1787756796.566162:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787756797.014049:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756797.014062:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787756797.014067:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756797.014080:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9853 00000100:00080000:0.0:1787756797.014092:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756797.014101:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787756797.014155:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588654795904/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756797.014203:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756797.014210:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787756797.014214:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756797.014223:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9853 00000100:00080000:0.0:1787756797.014232:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756797.014242:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756797.014258:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756797.014282:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654795904:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787756797.014287:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588654796032/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756797.014309:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756797.014314:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787756797.014317:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756797.014323:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9853 00000100:00080000:0.0:1787756797.014329:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756797.014335:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756797.014342:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756797.014359:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588654796160/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756797.014373:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787756797.014381:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787756797.014383:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787756797.014389:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756797.014397:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9853 00000100:00080000:0.0:1787756797.014404:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756797.014411:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756797.014418:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787756797.014434:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588654796288/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756797.014447:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756797/real 1787756797] req@ffff8c05a9f77100 x1874588654795904/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756797.014451:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756797.014471:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787756797.014477:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756797.014487:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9853 00000100:00080000:0.0:1787756797.014493:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756797.014497:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756797.014503:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756797.014513:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588654795904/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756797.014522:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588654796416/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756797.014561:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756797.014564:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756797.014567:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787756797.014574:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756797.014585:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787756797.014586:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9853 00000100:00080000:0.0:1787756797.014600:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787756797.014608:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654795904:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787756797.014610:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756797.014621:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756797.014637:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588654796544/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756797.014648:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756797.014652:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787756797.014655:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756797.014661:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9853 00000100:00100000:2.0:1787756797.014668:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654796032:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787756797.014670:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756797.014686:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756797.014693:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787756797.014718:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588654796672/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787756797.014729:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756797.014753:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756797/real 1787756797] req@ffff8c05a9f74a80 x1874588654796032/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756797.014784:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787756797.014791:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588654796032/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756797.014800:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787756797.014808:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756797.014824:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9853 00000100:00080000:0.0:1787756797.014837:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756797.014842:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756797.014849:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:2.0:1787756797.014858:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000008:00100000:0.0:1787756797.014866:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756797.014867:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654796032:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756797.014970:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588654796800/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756797.014971:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654796160:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756797.015033:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756797.015061:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756797/real 1787756797] req@ffff8c05a9f75c00 x1874588654796160/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756797.015088:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588654796160/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756797.015106:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756797.015113:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756797.015118:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654796160:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756797.015127:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654796288:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756797.015157:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756797.015172:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756797/real 1787756797] req@ffff8c05a9f74380 x1874588654796288/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756797.015188:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588654796288/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756797.015198:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756797.015202:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756797.015207:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654796288:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756797.015214:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654796416:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756797.015234:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756797.015245:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756797/real 1787756797] req@ffff8c05a9f76a00 x1874588654796416/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756797.015258:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588654796416/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756797.015269:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756797.015272:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756797.015276:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654796416:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756797.015282:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654796544:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756797.015309:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756797.015320:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756797/real 1787756797] req@ffff8c05a9f76d80 x1874588654796544/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756797.015334:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588654796544/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756797.015343:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756797.015347:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756797.015350:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654796544:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756797.015374:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654796672:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756797.015394:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756797.015406:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756797/real 1787756797] req@ffff8c05a9f77b80 x1874588654796672/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756797.015421:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588654796672/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756797.015431:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756797.015434:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756797.015438:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654796672:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756797.015444:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654796800:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756797.015472:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756797.015482:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756797/real 1787756797] req@ffff8c05a9f76680 x1874588654796800/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756797.015496:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588654796800/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756797.015506:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756797.015509:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756797.015512:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654796800:192.168.201.107@tcp:8:kworker.0 00010000:00010000:3.0:1787756798.614086:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787756799.638195:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787756802.133971:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756802.133979:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787756802.133983:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756802.133991:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9858 00000100:00080000:0.0:1787756802.134000:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756802.134006:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787756802.134070:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588654796928/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756802.134105:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756802.134109:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787756802.134113:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756802.134119:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9858 00000100:00080000:0.0:1787756802.134126:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756802.134132:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756802.134142:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756802.134161:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588654797056/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756802.134176:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756802.134181:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787756802.134184:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756802.134189:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9858 00000100:00080000:0.0:1787756802.134195:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756802.134200:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756802.134209:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756802.134223:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654796928:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787756802.134231:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588654797184/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756802.134254:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756802.134259:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787756802.134261:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756802.134269:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9858 00000100:00080000:0.0:1787756802.134274:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756802.134279:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756802.134288:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787756802.134322:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588654797312/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756802.134333:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756802.134338:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787756802.134341:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756802.134349:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9858 00000800:00000100:2.0:1787756802.134356:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787756802.134362:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756802.134373:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756802.134381:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756802.134399:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588654797440/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756802.134409:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756802.134412:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787756802.134414:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756802.134417:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9858 00000100:00080000:0.0:1787756802.134420:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756802.134422:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756802.134426:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756802.134436:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756802/real 1787756802] req@ffff8c05a9f76680 x1874588654796928/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756802.134438:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588654797568/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756802.134513:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588654796928/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756802.134515:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756802.134537:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787756802.134545:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756802.134553:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756802.134554:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9858 00000100:00080000:0.0:1787756802.134573:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756802.134574:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787756802.134588:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756802.134590:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654796928:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787756802.134609:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654797056:192.168.201.107@tcp:38:kworker.0 00000008:00100000:0.0:1787756802.134610:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787756802.134644:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588654797696/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787756802.134656:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787756802.134676:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756802.134681:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787756802.134682:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756802/real 1787756802] req@ffff8c05a9f77b80 x1874588654797056/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756802.134685:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756802.134713:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9858 00000100:00100000:2.0:1787756802.134714:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588654797056/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756802.134730:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756802.134733:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756802.134741:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787756802.134747:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756802.134752:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654797056:192.168.201.107@tcp:38:kworker.0 00000008:00100000:0.0:1787756802.134759:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787756802.134778:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588654797824/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756802.134825:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654797184:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756802.134872:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756802.134944:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756802/real 1787756802] req@ffff8c05a9f76d80 x1874588654797184/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756802.134959:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588654797184/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756802.134968:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756802.134972:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756802.134976:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654797184:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756802.134984:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654797312:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756802.135012:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756802.135023:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756802/real 1787756802] req@ffff8c05a9f76a00 x1874588654797312/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756802.135036:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588654797312/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756802.135045:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756802.135048:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756802.135051:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654797312:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756802.135057:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654797440:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756802.135075:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756802.135085:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756802/real 1787756802] req@ffff8c05a9f74380 x1874588654797440/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756802.135097:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588654797440/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756802.135106:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756802.135109:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756802.135112:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654797440:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756802.135118:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654797568:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756802.135134:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756802.135144:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756802/real 1787756802] req@ffff8c05a9f75c00 x1874588654797568/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756802.135157:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588654797568/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756802.135165:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756802.135168:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756802.135171:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654797568:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756802.135177:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654797696:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756802.135194:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756802.135203:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756802/real 1787756802] req@ffff8c05a9f74a80 x1874588654797696/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756802.135215:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588654797696/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756802.135224:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756802.135227:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756802.135230:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654797696:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756802.135235:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654797824:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756802.135252:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756802.135261:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756802/real 1787756802] req@ffff8c05a9f77100 x1874588654797824/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756802.135273:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588654797824/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756802.135289:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756802.135293:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756802.135296:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654797824:192.168.201.107@tcp:8:kworker.0 00010000:00010000:3.0:1787756804.758095:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756806.806126:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756807.254078:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756807.254090:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787756807.254102:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756807.254117:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9863 00000100:00080000:3.0:1787756807.254132:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756807.254141:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787756807.254207:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588654797952/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756807.254234:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756807.254238:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756807.254240:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756807.254244:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9863 00000100:00080000:3.0:1787756807.254249:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756807.254253:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756807.254276:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756807.254289:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588654798080/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787756807.254308:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654797952:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787756807.254308:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756807.254319:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756807.254331:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756807.254341:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9863 00000100:00080000:3.0:1787756807.254347:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756807.254354:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756807.254362:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756807.254381:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588654798208/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756807.254399:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756807.254404:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756807.254406:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756807.254412:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9863 00000100:00080000:3.0:1787756807.254417:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756807.254422:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756807.254429:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756807.254444:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588654798336/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756807.254456:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756807.254462:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756807.254464:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756807.254479:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9863 00000100:00080000:3.0:1787756807.254483:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756807.254491:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756807.254500:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756807.254515:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79c00 x1874588654798464/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787756807.254532:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654798080:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787756807.254533:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756807.254556:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756807.254561:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756807.254570:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9863 00000100:00080000:3.0:1787756807.254576:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756807.254580:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756807.254589:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756807.254610:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588654798592/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756807.254622:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756807.254630:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787756807.254634:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654798208:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787756807.254635:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756807.254655:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9863 00000100:00080000:3.0:1787756807.254665:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756807.254670:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756807.254677:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756807.254715:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588654798720/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756807.254730:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756807.254734:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756807.254737:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756807.254742:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9863 00000100:00080000:3.0:1787756807.254747:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756807.254752:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756807.254759:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787756807.254764:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654798336:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787756807.254784:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588654798848/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787756807.254845:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654798464:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756807.254972:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654798592:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756807.255016:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654798720:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756807.255056:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654798848:192.168.201.107@tcp:8:kworker.0 00010000:00010000:0.0:1787756807.830104:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787756808.278283:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.107:988 00000400:00000100:2.0:1787756808.278334:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.107@tcp at host 192.168.201.107 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787756808.278354:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787756808.278399:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787756808.278410:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787756808.278419:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787756808.278429:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787756808.278440:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787756808.278452:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787756808.278473:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:0.0:1787756808.278494:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756807/real 1787756808] req@ffff8c05a9f79c00 x1874588654798464/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756808.278542:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79c00 x1874588654798464/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756808.278563:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756808.278571:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756808.278578:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654798464:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756808.278606:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756807/real 1787756808] req@ffff8c05a9f7b480 x1874588654798592/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756808.278630:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588654798592/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756808.278645:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756808.278652:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756808.278659:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654798592:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756808.278671:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756807/real 1787756808] req@ffff8c05a9f78a80 x1874588654798720/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756808.278718:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588654798720/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756808.278734:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756808.278741:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756808.278749:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654798720:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756808.278762:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756807/real 1787756808] req@ffff8c05a9f7a680 x1874588654798848/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756808.278783:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588654798848/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756808.278797:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756808.278802:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756808.278808:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654798848:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756808.278819:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756807/real 1787756808] req@ffff8c05a9f78380 x1874588654798080/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756808.278841:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588654798080/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756808.278857:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756808.278862:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756808.278868:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654798080:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756808.278964:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756807/real 1787756808] req@ffff8c05a9f79880 x1874588654798208/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756808.278992:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588654798208/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756808.279007:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756808.279013:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756808.279020:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654798208:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756808.279032:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756807/real 1787756808] req@ffff8c05a9f7aa00 x1874588654798336/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756808.279056:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588654798336/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756808.279072:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756808.279080:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756808.279087:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654798336:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756808.279103:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756807/real 1787756808] req@ffff8c05a9f7a300 x1874588654797952/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756808.279126:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588654797952/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756808.279139:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756808.279144:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756808.279150:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654797952:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787756812.374103:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756812.374120:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787756812.374127:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756812.374144:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9868 00000100:00080000:0.0:1787756812.374158:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756812.374169:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787756812.374214:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588654798976/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756812.374256:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756812.374261:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787756812.374263:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756812.374280:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9868 00000100:00080000:0.0:1787756812.374285:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756812.374292:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756812.374304:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756812.374325:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588654799104/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756812.374336:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654798976:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787756812.374365:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756812.374373:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787756812.374378:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756812.374388:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9868 00000100:00080000:0.0:1787756812.374397:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756812.374406:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756812.374417:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756812.374462:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588654799232/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787756812.374476:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787756812.374499:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756812.374508:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787756812.374513:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756812.374524:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9868 00000100:00080000:0.0:1787756812.374535:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756812.374546:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756812.374556:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756812/real 1787756812] req@ffff8c05a9f75180 x1874588654798976/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787756812.374562:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787756812.374601:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588654799360/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756812.374648:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787756812.374652:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588654798976/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756812.374665:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787756812.374674:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756812.374680:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756812.374688:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9868 00000100:00080000:2.0:1787756812.374696:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787756812.374701:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787756812.374708:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654798976:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787756812.374716:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756812.374728:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756812.374749:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588654799488/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756812.374756:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654799104:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787756812.374784:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756812.374790:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787756812.374793:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756812.374801:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9868 00000100:00080000:0.0:1787756812.374809:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756812.374814:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756812.374822:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787756812.374824:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756812.374848:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588654799616/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756812.374859:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756812/real 1787756812] req@ffff8c05a9f74000 x1874588654799104/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756812.374961:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787756812.374963:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588654799104/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756812.374978:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787756812.374987:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756812.375001:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9868 00000100:00080000:2.0:1787756812.375004:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756812.375010:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756812.375035:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787756812.375037:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756812.375051:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654799104:192.168.201.107@tcp:38:kworker.0 00000008:00100000:0.0:1787756812.375058:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756812.375067:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654799232:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756812.375100:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588654799744/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787756812.375119:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787756812.375134:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756812.375141:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787756812.375144:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787756812.375149:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756812/real 1787756812] req@ffff8c05a9f75880 x1874588654799232/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756812.375156:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9868 00000100:00080000:0.0:1787756812.375168:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756812.375181:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756812.375189:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588654799232/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787756812.375194:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787756812.375214:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756812.375220:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756812.375228:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588654799872/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756812.375234:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654799232:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756812.375259:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654799360:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756812.375291:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756812.375309:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756812/real 1787756812] req@ffff8c05a9f75500 x1874588654799360/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756812.375334:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588654799360/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756812.375344:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756812.375348:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756812.375352:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654799360:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756812.375375:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654799488:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756812.375409:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756812.375431:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756812/real 1787756812] req@ffff8c05a9f74e00 x1874588654799488/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756812.375457:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588654799488/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756812.375474:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756812.375480:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756812.375486:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654799488:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756812.375495:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654799616:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756812.375525:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756812.375546:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756812/real 1787756812] req@ffff8c05a9f76300 x1874588654799616/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756812.375566:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588654799616/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756812.375595:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756812.375602:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756812.375633:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654799616:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756812.375647:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654799744:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756812.375686:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756812.375703:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756812/real 1787756812] req@ffff8c05a9f77800 x1874588654799744/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756812.375718:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588654799744/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756812.375729:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756812.375734:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756812.375738:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654799744:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756812.375745:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654799872:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756812.375765:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756812.375777:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756812/real 1787756812] req@ffff8c05a9f75180 x1874588654799872/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756812.375791:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588654799872/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756812.375801:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756812.375804:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756812.375808:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654799872:192.168.201.107@tcp:8:kworker.0 00010000:00010000:3.0:1787756812.950116:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756814.998144:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787756816.022242:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756817.493944:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756817.493952:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787756817.493955:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756817.493963:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9873 00000100:00080000:2.0:1787756817.493972:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756817.493978:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756817.494010:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588654800000/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756817.494033:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756817.494035:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756817.494037:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756817.494040:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9873 00000100:00080000:2.0:1787756817.494042:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756817.494045:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756817.494052:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756817.494061:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654800000:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787756817.494064:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588654800128/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756817.494074:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756817.494077:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756817.494078:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756817.494081:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9873 00000100:00080000:2.0:1787756817.494083:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756817.494087:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756817.494090:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756817.494099:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588654800256/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756817.494106:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756817.494109:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756817.494111:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756817.494113:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9873 00000100:00080000:2.0:1787756817.494115:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756817.494120:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:0.0:1787756817.494121:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:2.0:1787756817.494124:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756817.494134:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588654800384/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756817.494141:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756817.494143:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756817.494145:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756817.494147:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9873 00000100:00080000:2.0:1787756817.494150:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756817.494153:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756817.494156:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756817.494160:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756817/real 1787756817] req@ffff8c05b2e70000 x1874588654800000/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756817.494182:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588654800000/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756817.494189:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588654800512/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756817.494201:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756817.494201:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756817.494209:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756817.494210:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756817.494212:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787756817.494216:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9873 00000100:00080000:2.0:1787756817.494220:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787756817.494222:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654800000:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787756817.494225:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756817.494233:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756817.494241:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588654800640/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756817.494247:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756817.494250:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787756817.494251:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756817.494255:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9873 00000100:00100000:0.0:1787756817.494256:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654800128:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787756817.494258:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756817.494263:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756817.494268:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756817.494278:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588654800768/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756817.494285:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787756817.494291:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787756817.494291:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756817.494294:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756817.494299:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9873 00000100:00080000:2.0:1787756817.494303:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756817.494308:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787756817.494309:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756817/real 1787756817] req@ffff8c05b2e70700 x1874588654800128/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787756817.494316:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787756817.494329:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588654800128/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756817.494331:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588654800896/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756817.494354:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756817.494359:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756817.494362:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654800128:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756817.494368:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654800256:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756817.494383:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756817.494390:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756817/real 1787756817] req@ffff8c05b2e73480 x1874588654800256/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756817.494398:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588654800256/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756817.494408:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756817.494410:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756817.494413:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654800256:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756817.494416:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654800384:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756817.494426:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756817.494432:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756817/real 1787756817] req@ffff8c05b2e71500 x1874588654800384/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756817.494439:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588654800384/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756817.494445:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756817.494447:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756817.494449:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654800384:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756817.494452:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654800512:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756817.494463:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756817.494469:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756817/real 1787756817] req@ffff8c05b2e73b80 x1874588654800512/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756817.494476:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588654800512/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756817.494481:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756817.494483:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756817.494486:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654800512:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756817.494489:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654800640:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756817.494499:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756817.494504:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756817/real 1787756817] req@ffff8c05b2e71c00 x1874588654800640/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756817.494511:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588654800640/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756817.494540:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756817.494542:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756817.494544:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654800640:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756817.494559:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654800768:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756817.494574:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756817.494581:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756817/real 1787756817] req@ffff8c05b2e71880 x1874588654800768/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756817.494589:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588654800768/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756817.494594:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756817.494597:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756817.494598:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654800768:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756817.494602:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654800896:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756817.494613:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756817.494623:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756817/real 1787756817] req@ffff8c05b2e70380 x1874588654800896/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756817.494635:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588654800896/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756817.494644:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756817.494647:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756817.494650:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654800896:192.168.201.107@tcp:8:kworker.0 00010000:00010000:3.0:1787756821.142044:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787756822.614230:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654801024:192.168.201.107@tcp:250:kworker.0 00000800:00000100:0.0:1787756822.614315:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756822.614391:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756822/real 1787756822] req@ffff8c05b2d9b480 x1874588654801024/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756822.614468:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654801024/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756822.614488:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756822.614495:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756822.614503:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654801024:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787756822.614537:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654801152:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756822.614577:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756822.614598:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756822/real 1787756822] req@ffff8c05b2d9b100 x1874588654801152/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756822.614621:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654801152/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756822.614649:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756822.614655:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756822.614662:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654801152:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756822.614674:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654801280:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756822.614704:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756822.614721:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756822/real 1787756822] req@ffff8c05b2d98700 x1874588654801280/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756822.614744:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654801280/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756822.614759:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756822.614766:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756822.614772:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654801280:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756822.614783:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654801408:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756822.614820:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756822.614851:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756822/real 1787756822] req@ffff8c05b2d9a300 x1874588654801408/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756822.614866:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588654801408/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756822.614941:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756822.614966:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756822.614975:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654801408:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756822.614987:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654801536:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756822.615031:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756822.615054:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756822/real 1787756822] req@ffff8c05b2d98380 x1874588654801536/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756822.615094:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654801536/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756822.615104:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756822.615108:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756822.615112:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654801536:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756822.615135:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654801664:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756822.615160:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756822.615173:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756822/real 1787756822] req@ffff8c05b2d99880 x1874588654801664/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756822.615186:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654801664/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756822.615195:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756822.615198:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756822.615201:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654801664:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756822.615208:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654801792:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756822.615224:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756822.615235:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756822/real 1787756822] req@ffff8c05b2d99f80 x1874588654801792/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756822.615248:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654801792/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756822.615258:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756822.615261:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756822.615264:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654801792:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756822.615269:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654801920:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756822.615285:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756822.615295:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756822/real 1787756822] req@ffff8c05b2d98000 x1874588654801920/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756822.615308:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654801920/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756822.615317:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756822.615319:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756822.615323:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654801920:192.168.201.107@tcp:8:kworker.0 00010000:00010000:3.0:1787756823.190153:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787756824.214203:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787756827.734301:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654802048:192.168.201.107@tcp:250:kworker.0 00000800:00000100:0.0:1787756827.734449:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756827.734542:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756827/real 1787756827] req@ffff8c05b2d9aa00 x1874588654802048/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756827.734624:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588654802048/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756827.734658:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756827.734675:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756827.734696:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654802048:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787756827.734742:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654802176:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756827.734816:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756827.734851:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756827/real 1787756827] req@ffff8c05b2d9bb80 x1874588654802176/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756827.734880:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588654802176/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756827.734900:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756827.734905:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756827.734970:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654802176:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756827.734985:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654802304:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756827.735043:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756827.735066:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756827/real 1787756827] req@ffff8c05b2d98000 x1874588654802304/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756827.735086:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654802304/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756827.735097:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756827.735101:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756827.735105:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654802304:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756827.735114:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654802432:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756827.735140:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756827.735153:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756827/real 1787756827] req@ffff8c05b2d99f80 x1874588654802432/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756827.735167:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654802432/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756827.735177:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756827.735181:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756827.735185:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654802432:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756827.735209:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654802560:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756827.735228:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756827.735239:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756827/real 1787756827] req@ffff8c05b2d99880 x1874588654802560/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756827.735251:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654802560/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756827.735260:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756827.735280:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756827.735283:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654802560:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756827.735308:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654802688:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756827.735329:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756827.735393:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756827/real 1787756827] req@ffff8c05b2d98380 x1874588654802688/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756827.735417:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654802688/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756827.735432:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756827.735438:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756827.735445:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654802688:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756827.735456:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654802816:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756827.735507:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756827.735522:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756827/real 1787756827] req@ffff8c05b2d9a300 x1874588654802816/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756827.735537:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588654802816/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756827.735548:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756827.735551:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756827.735556:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654802816:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756827.735565:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654802944:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756827.735598:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756827.735634:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756827/real 1787756827] req@ffff8c05b2d98700 x1874588654802944/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756827.735657:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654802944/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756827.735684:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756827.735688:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756827.735692:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654802944:192.168.201.107@tcp:8:kworker.0 00010000:00010000:3.0:1787756829.334175:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756831.382092:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787756832.406171:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787756832.854073:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756832.854089:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787756832.854095:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756832.854111:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9889 00000100:00080000:0.0:1787756832.854128:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756832.854152:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787756832.854230:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588654803072/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756832.854318:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756832.854323:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787756832.854326:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756832.854334:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9889 00000100:00080000:0.0:1787756832.854341:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756832.854350:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756832.854367:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756832.854397:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588654803200/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756832.854398:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654803072:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787756832.854433:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756832.854440:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787756832.854444:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756832.854452:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9889 00000100:00080000:0.0:1787756832.854459:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756832.854464:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756832.854472:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756832.854488:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588654803328/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787756832.854518:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787756832.854529:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756832.854537:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787756832.854540:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756832.854549:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9889 00000100:00080000:0.0:1787756832.854567:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756832.854574:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756832.854585:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787756832.854600:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588654803456/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756832.854616:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756832/real 1787756832] req@ffff8c05a9f77100 x1874588654803072/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756832.854620:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756832.854640:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787756832.854643:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756832.854656:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9889 00000100:00080000:0.0:1787756832.854675:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756832.854702:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756832.854712:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756832.854717:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588654803072/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756832.854733:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588654803584/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756832.854781:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756832.854781:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756832.854801:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756832.854810:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787756832.854813:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756832.854833:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9889 00000100:00100000:2.0:1787756832.854834:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654803072:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787756832.854848:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756832.854862:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756832.854870:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756832.854967:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654803200:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756832.854992:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588654803712/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787756832.855021:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787756832.855028:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756832.855036:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787756832.855039:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756832.855062:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9889 00000100:00100000:2.0:1787756832.855064:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756832/real 1787756832] req@ffff8c05a9f74a80 x1874588654803200/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756832.855071:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756832.855087:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756832.855093:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588654803200/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787756832.855104:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787756832.855106:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756832.855117:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756832.855141:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654803200:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756832.855153:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588654803840/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756832.855159:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654803328:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787756832.855200:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787756832.855210:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787756832.855217:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787756832.855225:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756832.855238:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9889 00000100:00080000:0.0:1787756832.855280:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787756832.855284:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756832/real 1787756832] req@ffff8c05a9f75c00 x1874588654803328/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756832.855293:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756832.855314:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756832.855324:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588654803328/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756832.855334:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787756832.855341:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588654803968/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756832.855343:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756832.855360:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654803328:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756832.855392:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654803456:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756832.855424:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756832.855436:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756832/real 1787756832] req@ffff8c05a9f74380 x1874588654803456/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756832.855445:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588654803456/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756832.855452:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756832.855455:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756832.855458:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654803456:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756832.855462:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654803584:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756832.855475:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756832.855482:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756832/real 1787756832] req@ffff8c05a9f76a00 x1874588654803584/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756832.855520:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588654803584/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756832.855530:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756832.855534:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756832.855538:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654803584:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756832.855564:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654803712:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756832.855587:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756832.855600:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756832/real 1787756832] req@ffff8c05a9f76d80 x1874588654803712/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756832.855614:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588654803712/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756832.855624:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756832.855627:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756832.855631:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654803712:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756832.855639:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654803840:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756832.855657:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756832.855669:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756832/real 1787756832] req@ffff8c05a9f77b80 x1874588654803840/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756832.855691:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588654803840/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756832.855719:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756832.855726:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756832.855733:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654803840:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756832.855746:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654803968:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756832.855786:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756832.855803:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756832/real 1787756832] req@ffff8c05a9f76680 x1874588654803968/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756832.855819:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588654803968/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756832.855829:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756832.855834:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756832.855838:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654803968:192.168.201.107@tcp:8:kworker.0 00010000:00010000:3.0:1787756837.526197:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787756837.974322:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654804096:192.168.201.107@tcp:250:kworker.0 00000800:00000100:2.0:1787756837.974382:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756837.974418:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756837/real 1787756837] req@ffff8c05b2d9b100 x1874588654804096/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756837.974439:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654804096/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756837.974462:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756837.974470:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756837.974492:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654804096:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787756837.974525:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654804224:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756837.974559:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756837.974580:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756837/real 1787756837] req@ffff8c05b2d9b480 x1874588654804224/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756837.974618:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654804224/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756837.974634:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756837.974640:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756837.974646:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654804224:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756837.974656:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654804352:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756837.974677:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756837.974693:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756837/real 1787756837] req@ffff8c05b2d98700 x1874588654804352/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756837.974726:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654804352/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756837.974748:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756837.974753:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756837.974758:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654804352:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756837.974779:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654804480:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756837.974818:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756837.974833:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756837/real 1787756837] req@ffff8c05b2d9a300 x1874588654804480/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756837.974843:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588654804480/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756837.974850:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756837.974852:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756837.974855:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654804480:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756837.974867:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654804608:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756837.974905:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756837.974914:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756837/real 1787756837] req@ffff8c05b2d98380 x1874588654804608/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756837.974922:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654804608/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756837.974927:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756837.974930:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756837.974932:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654804608:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756837.974936:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654804736:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756837.974946:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756837.974952:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756837/real 1787756837] req@ffff8c05b2d99880 x1874588654804736/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756837.974959:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654804736/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756837.974969:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756837.974971:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756837.974973:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654804736:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756837.974976:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654804864:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756837.974985:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756837.974991:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756837/real 1787756837] req@ffff8c05b2d99f80 x1874588654804864/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756837.974998:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654804864/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756837.975004:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756837.975005:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756837.975007:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654804864:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756837.975010:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654804992:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756837.975020:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756837.975025:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756837/real 1787756837] req@ffff8c05b2d98000 x1874588654804992/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756837.975031:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654804992/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756837.975036:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756837.975039:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756837.975041:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654804992:192.168.201.107@tcp:8:kworker.0 00010000:00010000:3.0:1787756839.574208:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787756840.598305:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787756843.094052:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756843.094090:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787756843.094111:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756843.094129:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9899 00000100:00080000:0.0:1787756843.094145:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756843.094159:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787756843.094211:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588654805120/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756843.094256:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756843.094261:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787756843.094263:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756843.094269:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9899 00000100:00080000:0.0:1787756843.094275:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756843.094282:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756843.094295:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756843.094313:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588654805248/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756843.094342:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756843.094363:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787756843.094367:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756843.094378:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9899 00000100:00080000:0.0:1787756843.094388:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756843.094399:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756843.094431:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756843.094460:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588654805376/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756843.094480:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756843.094488:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787756843.094493:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756843.094507:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9899 00000100:00080000:0.0:1787756843.094520:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756843.094528:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756843.094540:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787756843.094566:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588654805504/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756843.094588:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756843.094595:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787756843.094604:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756843.094619:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9899 00000100:00080000:0.0:1787756843.094635:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756843.094646:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756843.094655:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756843.094689:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588654805632/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756843.094754:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756843.094769:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787756843.094774:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756843.094785:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9899 00000100:00080000:0.0:1787756843.094797:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756843.094814:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756843.094825:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756843.094851:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588654805760/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756843.094944:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756843.094956:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787756843.094968:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756843.094986:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9899 00000100:00080000:0.0:1787756843.095004:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756843.095022:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756843.095037:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787756843.095093:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588654805888/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756843.095138:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756843.095160:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787756843.095168:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756843.095182:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9899 00000100:00080000:0.0:1787756843.095196:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756843.095216:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756843.095231:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787756843.095264:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588654806016/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787756845.718261:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756847.766185:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787756848.214027:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756848.214040:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787756848.214045:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756848.214058:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9904 00000100:00080000:0.0:1787756848.214082:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756848.214092:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787756848.214140:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588654806144/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756848.214179:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756848.214184:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787756848.214187:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756848.214193:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9904 00000100:00080000:0.0:1787756848.214197:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756848.214201:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756848.214213:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756848.214231:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588654806272/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756848.214245:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756848.214250:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787756848.214253:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756848.214258:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9904 00000100:00080000:0.0:1787756848.214271:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756848.214282:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756848.214292:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756848.214308:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588654806400/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756848.214322:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756848.214327:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787756848.214329:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756848.214334:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9904 00000100:00080000:0.0:1787756848.214339:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756848.214344:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756848.214350:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787756848.214363:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588654806528/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756848.214384:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756848.214389:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787756848.214394:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756848.214406:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9904 00000100:00080000:0.0:1787756848.214414:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756848.214421:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756848.214429:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756848.214446:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588654806656/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756848.214458:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756848.214463:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787756848.214469:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756848.214477:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9904 00000100:00080000:0.0:1787756848.214488:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756848.214499:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756848.214510:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756848.214528:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588654806784/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756848.214561:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756848.214572:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787756848.214581:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756848.214596:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9904 00000100:00080000:0.0:1787756848.214606:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756848.214614:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756848.214629:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787756848.214650:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588654806912/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756848.214663:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756848.214674:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787756848.214678:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756848.214684:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9904 00000100:00080000:0.0:1787756848.214688:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756848.214697:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756848.214703:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787756848.214725:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588654807040/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787756848.790205:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756853.334030:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756853.334041:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787756853.334045:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756853.334056:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9909 00000100:00080000:3.0:1787756853.334068:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756853.334075:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787756853.334116:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588654807168/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756853.334147:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756853.334151:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756853.334153:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756853.334157:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9909 00000100:00080000:3.0:1787756853.334160:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756853.334163:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756853.334172:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756853.334186:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588654807296/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756853.334198:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756853.334202:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756853.334204:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756853.334208:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9909 00000100:00080000:3.0:1787756853.334212:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756853.334219:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756853.334225:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756853.334243:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588654807424/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756853.334256:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756853.334260:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756853.334262:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756853.334267:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9909 00000100:00080000:3.0:1787756853.334271:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756853.334274:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756853.334279:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756853.334293:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588654807552/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756853.334309:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756853.334328:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756853.334332:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756853.334340:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9909 00000100:00080000:3.0:1787756853.334348:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756853.334358:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756853.334371:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756853.334393:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588654807680/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756853.334434:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756853.334440:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756853.334443:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756853.334455:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9909 00000100:00080000:3.0:1787756853.334464:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756853.334472:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756853.334482:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756853.334514:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588654807808/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756853.334537:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756853.334541:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787756853.334548:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756853.334561:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9909 00000100:00080000:3.0:1787756853.334570:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756853.334578:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756853.334586:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756853.334608:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588654807936/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756853.334638:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756853.334642:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756853.334644:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756853.334650:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9909 00000100:00080000:3.0:1787756853.334653:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756853.334656:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756853.334662:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756853.334679:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588654808064/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787756853.335136:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588654808064/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787756853.335151:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787756853.335154:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787756853.335157:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654808064:192.168.201.107@tcp:8:kworker.0 00010000:00010000:3.0:1787756853.910189:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787756854.934077:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756855.958071:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787756856.982082:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787756858.006000:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787756858.453963:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756858.453974:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787756858.453978:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756858.453988:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9914 00000100:00080000:1.0:1787756858.454000:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756858.454007:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787756858.454050:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588654808192/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756858.454080:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756858.454083:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787756858.454085:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756858.454088:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9914 00000100:00080000:1.0:1787756858.454090:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756858.454093:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756858.454102:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787756858.454113:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588654808320/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756858.454120:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756858.454124:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787756858.454126:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756858.454129:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9914 00000100:00080000:1.0:1787756858.454131:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756858.454133:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756858.454137:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756858.454145:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654808192:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787756858.454148:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654808448/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756858.454163:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756858.454165:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787756858.454166:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756858.454169:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9914 00000100:00080000:1.0:1787756858.454171:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756858.454173:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756858.454176:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787756858.454185:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654808576/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756858.454191:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756858.454193:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787756858.454195:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756858.454197:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9914 00000100:00080000:1.0:1787756858.454200:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756858.454203:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756858.454207:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787756858.454215:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654808704/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756858.454222:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756858.454225:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787756858.454226:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756858.454229:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9914 00000100:00080000:1.0:1787756858.454231:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756858.454233:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756858.454236:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787756858.454242:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787756858.454246:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654808832/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756858.454253:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756858.454255:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787756858.454256:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756858.454260:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9914 00000100:00080000:1.0:1787756858.454269:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756858.454272:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756858.454277:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787756858.454290:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588654808960/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756858.454296:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756858/real 1787756858] req@ffff8c05b2d9bb80 x1874588654808192/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787756858.454314:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756858.454320:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787756858.454326:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787756858.454326:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588654808192/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787756858.454337:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9914 00000100:00080000:1.0:1787756858.454342:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756858.454343:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787756858.454347:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:2.0:1787756858.454353:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756858.454360:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654808192:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787756858.454369:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654808320:192.168.201.107@tcp:38:kworker.0 00000008:00100000:1.0:1787756858.454372:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787756858.454392:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588654809088/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787756858.454403:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756858.454422:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756858/real 1787756858] req@ffff8c05b2d9aa00 x1874588654808320/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756858.454432:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588654808320/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756858.454439:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756858.454441:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756858.454444:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654808320:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756858.454466:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654808448:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756858.454484:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756858.454496:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756858/real 1787756858] req@ffff8c05b2d98000 x1874588654808448/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756858.454509:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654808448/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756858.454518:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756858.454522:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756858.454525:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654808448:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756858.454532:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654808576:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756858.454555:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756858.454568:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756858/real 1787756858] req@ffff8c05b2d99f80 x1874588654808576/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756858.454581:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654808576/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756858.454589:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756858.454592:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756858.454596:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654808576:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756858.454602:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654808704:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756858.454622:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756858.454635:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756858/real 1787756858] req@ffff8c05b2d99880 x1874588654808704/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756858.454647:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654808704/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756858.454657:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756858.454661:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756858.454664:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654808704:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756858.454670:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654808832:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756858.454691:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756858.454702:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756858/real 1787756858] req@ffff8c05b2d98380 x1874588654808832/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756858.454715:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654808832/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756858.454725:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756858.454728:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756858.454732:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654808832:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756858.454738:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654808960:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756858.454760:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756858.454772:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756858/real 1787756858] req@ffff8c05b2d9a300 x1874588654808960/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756858.454785:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588654808960/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756858.454807:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756858.454817:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756858.454821:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654808960:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756858.454826:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654809088:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756858.454842:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756858.454850:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756858/real 1787756858] req@ffff8c05b2d98700 x1874588654809088/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756858.454858:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654809088/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756858.454864:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756858.454866:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756858.454868:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654809088:192.168.201.107@tcp:8:kworker.0 00010000:00010000:1.0:1787756859.030089:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787756860.054002:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787756861.078054:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756862.101993:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787756863.126072:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787756863.573941:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756863.573950:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787756863.573953:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756863.573962:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9919 00000100:00080000:1.0:1787756863.573983:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756863.573989:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787756863.574018:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588654809216/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756863.574040:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756863.574043:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787756863.574044:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756863.574047:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9919 00000100:00080000:1.0:1787756863.574049:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756863.574051:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756863.574059:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787756863.574068:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588654809344/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756863.574076:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756863.574078:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787756863.574080:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756863.574082:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9919 00000100:00080000:1.0:1787756863.574084:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756863.574086:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756863.574090:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787756863.574096:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588654809472/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756863.574104:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756863.574106:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787756863.574108:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756863.574113:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9919 00000100:00100000:2.0:1787756863.574113:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654809216:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787756863.574116:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756863.574121:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756863.574127:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787756863.574135:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588654809600/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756863.574141:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756863.574143:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787756863.574145:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756863.574148:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9919 00000100:00080000:1.0:1787756863.574150:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756863.574153:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756863.574156:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787756863.574164:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654809728/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756863.574169:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756863.574171:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787756863.574173:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756863.574176:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9919 00000100:00080000:1.0:1787756863.574179:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756863.574181:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756863.574184:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787756863.574188:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787756863.574193:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654809856/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756863.574200:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756863.574203:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787756863.574204:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756863.574208:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9919 00000100:00080000:1.0:1787756863.574210:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756863.574213:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756863.574216:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787756863.574224:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654809984/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756863.574230:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756863/real 1787756863] req@ffff8c05b2d9b480 x1874588654809216/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787756863.574234:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756863.574243:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787756863.574246:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756863.574252:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9919 00000100:00080000:1.0:1787756863.574255:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787756863.574259:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654809216/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787756863.574260:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756863.574266:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787756863.574278:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654810112/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756863.574279:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756863.574288:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756863.574295:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654809216:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787756863.574302:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654809344:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756863.574322:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756863.574331:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756863/real 1787756863] req@ffff8c05b2d9b100 x1874588654809344/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756863.574339:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654809344/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756863.574345:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756863.574347:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756863.574349:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654809344:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756863.574368:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654809472:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756863.574379:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756863.574385:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756863/real 1787756863] req@ffff8c05b2d98700 x1874588654809472/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756863.574393:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654809472/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756863.574398:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756863.574399:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756863.574401:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654809472:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756863.574405:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654809600:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756863.574418:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756863.574428:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756863/real 1787756863] req@ffff8c05b2d9a300 x1874588654809600/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756863.574440:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588654809600/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756863.574449:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756863.574453:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756863.574456:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654809600:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756863.574462:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654809728:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756863.574484:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756863.574497:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756863/real 1787756863] req@ffff8c05b2d98380 x1874588654809728/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756863.574510:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654809728/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756863.574516:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756863.574519:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756863.574521:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654809728:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756863.574525:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654809856:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756863.574540:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756863.574548:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756863/real 1787756863] req@ffff8c05b2d99880 x1874588654809856/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756863.574556:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654809856/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756863.574561:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756863.574563:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756863.574565:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654809856:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756863.574569:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654809984:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756863.574579:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756863.574585:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756863/real 1787756863] req@ffff8c05b2d99f80 x1874588654809984/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756863.574592:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654809984/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756863.574597:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756863.574599:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756863.574601:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654809984:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756863.574604:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654810112:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756863.574616:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756863.574626:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756863/real 1787756863] req@ffff8c05b2d98000 x1874588654810112/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756863.574639:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654810112/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756863.574648:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756863.574651:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756863.574655:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654810112:192.168.201.107@tcp:8:kworker.0 00010000:00010000:3.0:1787756864.150078:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787756865.174091:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787756866.198050:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787756867.222051:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787756868.246024:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787756868.693949:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756868.693965:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787756868.693969:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756868.693977:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9925 00000100:00080000:1.0:1787756868.693986:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756868.693991:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787756868.694022:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588654810240/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756868.694050:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756868.694053:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787756868.694055:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756868.694057:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9925 00000100:00080000:1.0:1787756868.694060:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756868.694063:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756868.694070:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787756868.694080:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588654810368/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756868.694088:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756868.694091:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787756868.694092:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756868.694095:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9925 00000100:00080000:1.0:1787756868.694097:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787756868.694100:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654810240:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787756868.694101:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756868.694108:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787756868.694116:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654810496/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756868.694124:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756868.694126:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787756868.694127:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756868.694130:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9925 00000100:00080000:1.0:1787756868.694133:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756868.694135:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756868.694139:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787756868.694147:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654810624/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756868.694154:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756868.694156:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787756868.694158:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756868.694160:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9925 00000100:00080000:1.0:1787756868.694162:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756868.694164:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756868.694167:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787756868.694175:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654810752/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756868.694182:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756868.694184:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787756868.694187:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787756868.694188:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654810368:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787756868.694193:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9925 00000100:00080000:1.0:1787756868.694197:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756868.694200:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756868.694203:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787756868.694212:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654810880/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756868.694219:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756868.694221:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787756868.694223:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756868.694227:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9925 00000100:00080000:1.0:1787756868.694230:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756868.694233:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756868.694237:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756868.694240:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654810496:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787756868.694249:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588654811008/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756868.694258:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756868.694261:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787756868.694262:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756868.694265:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9925 00000100:00080000:1.0:1787756868.694268:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787756868.694269:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654810624:192.168.201.107@tcp:8:kworker.0 00000100:00080000:1.0:1787756868.694274:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756868.694279:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787756868.694289:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588654811136/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756868.694296:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654810752:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756868.694313:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654810880:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756868.694331:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654811008:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756868.694344:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654811136:192.168.201.107@tcp:8:kworker.0 00010000:00010000:1.0:1787756869.270080:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756870.294015:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787756871.318051:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787756871.766097:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.107:988 00000400:00000100:0.0:1787756871.766135:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.107@tcp at host 192.168.201.107 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787756871.766148:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787756871.766174:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787756871.766186:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787756871.766200:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:2.0:1787756871.766200:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756868/real 1787756871] req@ffff8c05b2d9aa00 x1874588654810240/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787756871.766216:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:2.0:1787756871.766233:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588654810240/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787756871.766236:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00080000:2.0:1787756871.766251:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000800:00000100:0.0:1787756871.766257:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00080000:2.0:1787756871.766259:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756871.766265:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654810240:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787756871.766271:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756868/real 1787756871] req@ffff8c05b2d9bb80 x1874588654810368/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787756871.766280:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:2.0:1787756871.766294:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588654810368/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756871.766306:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756871.766311:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756871.766317:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654810368:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756871.766343:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756868/real 1787756871] req@ffff8c05b2d9a300 x1874588654811008/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756871.766356:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588654811008/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756871.766365:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756871.766370:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756871.766375:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654811008:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756871.766383:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756868/real 1787756871] req@ffff8c05b2d98700 x1874588654811136/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756871.766399:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654811136/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756871.766408:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756871.766411:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756871.766414:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654811136:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756871.766420:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756868/real 1787756871] req@ffff8c05b2d98000 x1874588654810496/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756871.766432:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654810496/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756871.766440:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756871.766444:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756871.766448:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654810496:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756871.766454:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756868/real 1787756871] req@ffff8c05b2d99f80 x1874588654810624/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756871.766467:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654810624/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756871.766475:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756871.766479:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756871.766482:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654810624:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756871.766489:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756868/real 1787756871] req@ffff8c05b2d99880 x1874588654810752/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756871.766501:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654810752/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756871.766509:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756871.766512:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756871.766516:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654810752:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756871.766522:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756868/real 1787756871] req@ffff8c05b2d98380 x1874588654810880/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756871.766533:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654810880/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756871.766541:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756871.766544:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756871.766547:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654810880:192.168.201.107@tcp:38:kworker.0 00010000:00010000:3.0:1787756872.342028:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787756873.366129:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787756873.814006:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756873.814015:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787756873.814019:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756873.814026:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9930 00000100:00080000:1.0:1787756873.814036:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756873.814042:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787756873.814073:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588654811264/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756873.814103:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756873.814107:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787756873.814109:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756873.814113:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9930 00000100:00080000:1.0:1787756873.814117:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756873.814121:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756873.814130:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787756873.814144:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588654811392/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756873.814155:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756873.814159:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787756873.814162:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756873.814167:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9930 00000100:00080000:1.0:1787756873.814170:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756873.814174:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756873.814179:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756873.814203:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654811264:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787756873.814205:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654811520/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756873.814222:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756873.814225:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787756873.814228:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756873.814232:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9930 00000100:00080000:1.0:1787756873.814244:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756873.814249:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756873.814255:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787756873.814268:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654811648/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756873.814278:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756873.814282:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787756873.814284:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787756873.814290:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787756873.814294:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9930 00000100:00080000:1.0:1787756873.814301:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756873.814306:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756873.814317:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787756873.814332:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654811776/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756873.814343:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787756873.814344:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756873/real 1787756873] req@ffff8c05b2d9b100 x1874588654811264/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787756873.814351:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787756873.814355:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756873.814365:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9930 00000100:00080000:1.0:1787756873.814372:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756873.814379:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756873.814380:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654811264/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787756873.814387:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787756873.814398:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787756873.814406:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654811904/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756873.814406:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756873.814415:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654811264:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787756873.814430:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654811392:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787756873.814438:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756873.814442:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787756873.814445:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756873.814450:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9930 00000100:00080000:1.0:1787756873.814455:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756873.814460:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756873.814467:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787756873.814473:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787756873.814484:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588654812032/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756873.814493:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756873/real 1787756873] req@ffff8c05b2d9b480 x1874588654811392/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787756873.814497:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756873.814506:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787756873.814510:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756873.814537:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9930 00000100:00100000:2.0:1787756873.814538:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654811392/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787756873.814545:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756873.814555:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:2.0:1787756873.814555:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756873.814563:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000008:00100000:1.0:1787756873.814565:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756873.814568:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654811392:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787756873.814583:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588654812160/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756873.814604:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654811520:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756873.814631:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756873.814644:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756873/real 1787756873] req@ffff8c05b2d98380 x1874588654811520/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756873.814655:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654811520/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756873.814663:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756873.814666:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756873.814670:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654811520:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756873.814676:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654811648:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756873.814695:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756873.814706:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756873/real 1787756873] req@ffff8c05b2d99880 x1874588654811648/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756873.814717:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654811648/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756873.814725:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756873.814728:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756873.814732:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654811648:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756873.814737:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654811776:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756873.814757:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756873.814768:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756873/real 1787756873] req@ffff8c05b2d99f80 x1874588654811776/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756873.814782:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654811776/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756873.814791:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756873.814795:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756873.814798:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654811776:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756873.814805:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654811904:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756873.814826:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756873.814840:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756873/real 1787756873] req@ffff8c05b2d98000 x1874588654811904/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756873.814853:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654811904/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756873.814862:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756873.814866:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756873.814869:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654811904:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756873.814914:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654812032:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756873.814936:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756873.814948:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756873/real 1787756873] req@ffff8c05b2d98700 x1874588654812032/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756873.814956:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654812032/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756873.814962:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756873.814964:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756873.814967:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654812032:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756873.814971:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654812160:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756873.814989:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756873.815000:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756873/real 1787756873] req@ffff8c05b2d9a300 x1874588654812160/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756873.815012:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588654812160/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756873.815020:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756873.815024:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756873.815027:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654812160:192.168.201.107@tcp:8:kworker.0 00010000:00010000:1.0:1787756874.390002:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787756875.414043:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787756876.438033:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787756877.462075:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756878.485999:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787756878.934012:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756878.934019:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787756878.934022:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756878.934030:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9935 00000100:00080000:1.0:1787756878.934039:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756878.934045:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787756878.934076:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588654812288/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756878.934106:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756878.934109:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787756878.934111:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756878.934113:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9935 00000100:00080000:1.0:1787756878.934116:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756878.934120:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756878.934127:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787756878.934138:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588654812416/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756878.934146:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756878.934149:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787756878.934151:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756878.934156:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9935 00000100:00080000:1.0:1787756878.934160:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787756878.934161:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654812288:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787756878.934165:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756878.934173:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787756878.934184:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588654812544/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756878.934193:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756878.934196:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787756878.934198:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756878.934202:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9935 00000100:00080000:1.0:1787756878.934221:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756878.934223:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756878.934227:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787756878.934233:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787756878.934236:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588654812672/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756878.934244:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756878.934247:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787756878.934249:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756878.934251:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9935 00000100:00080000:1.0:1787756878.934254:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756878.934256:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756878.934260:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787756878.934269:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654812800/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787756878.934275:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756878/real 1787756878] req@ffff8c05b2d9bb80 x1874588654812288/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787756878.934279:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756878.934287:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787756878.934290:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756878.934298:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9935 00000100:00080000:1.0:1787756878.934300:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756878.934303:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756878.934307:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756878.934312:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588654812288/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787756878.934317:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654812928/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756878.934345:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756878.934348:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787756878.934349:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787756878.934349:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756878.934357:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9935 00000100:00080000:0.0:1787756878.934361:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787756878.934361:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756878.934368:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756878.934372:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787756878.934373:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654812288:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787756878.934381:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654813056/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787756878.934388:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654812416:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787756878.934390:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756878.934397:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787756878.934399:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756878.934402:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9935 00000100:00080000:1.0:1787756878.934405:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756878.934407:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756878.934410:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787756878.934419:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654813184/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787756878.934443:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756878.934466:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756878/real 1787756878] req@ffff8c05b2d9aa00 x1874588654812416/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756878.934483:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588654812416/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756878.934492:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756878.934497:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756878.934502:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654812416:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756878.934533:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654812544:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756878.934561:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756878.934581:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756878/real 1787756878] req@ffff8c05b2d9a300 x1874588654812544/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756878.934608:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588654812544/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756878.934617:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756878.934621:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756878.934625:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654812544:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756878.934632:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654812672:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756878.934652:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756878.934664:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756878/real 1787756878] req@ffff8c05b2d98700 x1874588654812672/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756878.934676:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654812672/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756878.934682:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756878.934684:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756878.934686:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654812672:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756878.934690:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654812800:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756878.934704:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756878.934711:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756878/real 1787756878] req@ffff8c05b2d98000 x1874588654812800/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756878.934719:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654812800/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756878.934728:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756878.934730:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756878.934732:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654812800:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756878.934735:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654812928:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756878.934745:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756878.934750:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756878/real 1787756878] req@ffff8c05b2d99f80 x1874588654812928/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756878.934758:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654812928/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756878.934763:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756878.934765:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756878.934766:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654812928:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756878.934769:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654813056:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756878.934779:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756878.934784:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756878/real 1787756878] req@ffff8c05b2d99880 x1874588654813056/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756878.934791:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654813056/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756878.934796:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756878.934798:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756878.934800:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654813056:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756878.934804:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654813184:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756878.934814:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756878.934819:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756878/real 1787756878] req@ffff8c05b2d98380 x1874588654813184/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756878.934826:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654813184/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756878.934831:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756878.934833:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756878.934834:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654813184:192.168.201.107@tcp:8:kworker.0 00010000:00010000:1.0:1787756879.510083:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756880.534039:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756881.558164:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787756882.582024:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787756883.606107:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787756884.053968:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756884.053976:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787756884.053979:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756884.053988:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9940 00000100:00080000:0.0:1787756884.053997:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756884.054003:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787756884.054039:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588654813312/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756884.054064:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756884.054067:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787756884.054071:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756884.054074:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9940 00000100:00080000:0.0:1787756884.054077:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756884.054080:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756884.054087:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756884.054097:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588654813440/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756884.054105:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756884.054110:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787756884.054111:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654813312:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787756884.054112:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756884.054120:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9940 00000100:00080000:0.0:1787756884.054123:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756884.054125:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756884.054128:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756884.054137:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588654813568/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756884.054144:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756884.054147:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787756884.054148:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756884.054150:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9940 00000100:00080000:0.0:1787756884.054153:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756884.054155:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756884.054158:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787756884.054165:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588654813696/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756884.054172:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787756884.054173:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787756884.054176:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787756884.054179:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756884.054184:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9940 00000100:00080000:0.0:1787756884.054187:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756884.054191:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756884.054194:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756884.054202:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588654813824/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756884.054208:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756884.054210:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787756884.054211:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756884.054215:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9940 00000100:00080000:0.0:1787756884.054217:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756884.054219:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756884.054222:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756884.054224:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756884/real 1787756884] req@ffff8c05a9f74a80 x1874588654813312/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756884.054232:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588654813952/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756884.054241:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756884.054244:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787756884.054247:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787756884.054247:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588654813312/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756884.054257:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9940 00000100:00080000:0.0:1787756884.054262:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756884.054266:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:2.0:1787756884.054269:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787756884.054270:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787756884.054280:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756884.054286:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588654814080/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756884.054288:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654813312:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787756884.054305:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756884.054309:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787756884.054323:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756884.054329:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9940 00000100:00100000:2.0:1787756884.054333:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654813440:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787756884.054334:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756884.054365:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756884.054370:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787756884.054376:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756884.054392:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588654814208/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756884.054399:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756884/real 1787756884] req@ffff8c05a9f75c00 x1874588654813440/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756884.054411:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588654813440/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756884.054419:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756884.054422:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756884.054425:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654813440:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756884.054430:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654813568:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756884.054442:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756884.054449:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756884/real 1787756884] req@ffff8c05a9f74380 x1874588654813568/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756884.054456:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588654813568/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756884.054461:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756884.054463:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756884.054465:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654813568:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756884.054468:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654813696:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756884.054479:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756884.054485:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756884/real 1787756884] req@ffff8c05a9f76a00 x1874588654813696/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756884.054492:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588654813696/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756884.054497:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756884.054499:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756884.054501:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654813696:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756884.054504:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654813824:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756884.054514:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756884.054519:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756884/real 1787756884] req@ffff8c05a9f76d80 x1874588654813824/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756884.054526:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588654813824/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756884.054532:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756884.054533:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756884.054535:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654813824:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756884.054538:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654813952:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756884.054548:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756884.054553:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756884/real 1787756884] req@ffff8c05a9f77b80 x1874588654813952/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756884.054560:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588654813952/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756884.054565:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756884.054566:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756884.054569:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654813952:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756884.054572:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654814080:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756884.054582:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756884.054587:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756884/real 1787756884] req@ffff8c05a9f76680 x1874588654814080/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756884.054594:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588654814080/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756884.054599:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756884.054601:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756884.054603:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654814080:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756884.054616:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654814208:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756884.054626:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756884.054632:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756884/real 1787756884] req@ffff8c05a9f74a80 x1874588654814208/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756884.054639:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588654814208/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756884.054644:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756884.054646:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756884.054647:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654814208:192.168.201.107@tcp:8:kworker.0 00010000:00010000:1.0:1787756884.630030:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787756885.654117:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756886.678040:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787756887.702214:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756888.726063:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756889.173953:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756889.173962:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787756889.173966:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756889.173974:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9945 00000100:00080000:3.0:1787756889.173982:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756889.173988:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787756889.174021:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588654814336/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756889.174043:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756889.174045:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756889.174047:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756889.174049:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9945 00000100:00080000:3.0:1787756889.174052:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756889.174054:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756889.174061:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756889.174071:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588654814464/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756889.174083:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756889.174086:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756889.174087:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756889.174092:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9945 00000100:00080000:3.0:1787756889.174094:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756889.174097:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756889.174101:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756889.174108:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588654814592/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756889.174118:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756889.174121:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756889.174122:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756889.174126:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9945 00000100:00080000:3.0:1787756889.174130:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787756889.174137:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654814336:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787756889.174137:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756889.174144:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756889.174156:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588654814720/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756889.174163:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756889.174165:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756889.174166:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756889.174169:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9945 00000100:00080000:3.0:1787756889.174171:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756889.174174:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756889.174177:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756889.174183:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588654814848/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756889.174189:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756889.174191:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756889.174193:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756889.174196:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9945 00000100:00080000:3.0:1787756889.174198:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756889.174201:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756889.174203:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756889.174211:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588654814976/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756889.174216:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756889.174218:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787756889.174219:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756889.174222:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9945 00000100:00080000:3.0:1787756889.174225:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756889.174227:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756889.174230:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756889.174236:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588654815104/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756889.174246:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756889.174263:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756889.174265:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787756889.174274:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787756889.174276:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9945 00000100:00080000:3.0:1787756889.174282:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756889.174286:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756889.174290:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756889.174299:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588654815232/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756889.174344:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756889/real 1787756889] req@ffff8c05a9f79500 x1874588654814336/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756889.174366:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588654814336/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756889.174379:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756889.174384:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756889.174389:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654814336:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787756889.174399:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654814464:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756889.174423:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756889.174434:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756889/real 1787756889] req@ffff8c05a9f7b100 x1874588654814464/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756889.174442:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588654814464/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756889.174448:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756889.174451:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756889.174453:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654814464:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756889.174475:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654814592:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756889.174487:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756889.174494:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756889/real 1787756889] req@ffff8c05a9f7b800 x1874588654814592/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756889.174502:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588654814592/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756889.174512:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756889.174515:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756889.174517:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654814592:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756889.174521:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654814720:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756889.174532:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756889.174538:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756889/real 1787756889] req@ffff8c05a9f7bb80 x1874588654814720/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756889.174546:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588654814720/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756889.174552:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756889.174553:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756889.174555:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654814720:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756889.174559:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654814848:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756889.174569:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756889.174575:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756889/real 1787756889] req@ffff8c05a9f7ad80 x1874588654814848/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756889.174583:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588654814848/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756889.174588:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756889.174590:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756889.174592:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654814848:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756889.174597:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654814976:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756889.174608:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756889.174613:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756889/real 1787756889] req@ffff8c05a9f79f80 x1874588654814976/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756889.174621:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588654814976/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756889.174626:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756889.174628:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756889.174629:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654814976:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756889.174633:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654815104:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756889.174643:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756889.174649:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756889/real 1787756889] req@ffff8c05a9f79180 x1874588654815104/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756889.174655:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588654815104/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756889.174660:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756889.174663:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756889.174665:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654815104:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756889.174668:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654815232:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756889.174678:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756889.174684:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756889/real 1787756889] req@ffff8c05a9f78e00 x1874588654815232/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756889.174692:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588654815232/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756889.174697:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756889.174699:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756889.174701:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654815232:192.168.201.107@tcp:8:kworker.0 00010000:00010000:3.0:1787756889.750089:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787756890.774102:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787756891.798186:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787756892.823177:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787756893.846068:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787756894.293981:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756894.293991:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787756894.293995:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756894.294003:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9950 00000100:00080000:1.0:1787756894.294012:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756894.294018:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787756894.294050:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588654815360/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756894.294078:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756894.294081:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787756894.294082:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756894.294085:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9950 00000100:00080000:1.0:1787756894.294087:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756894.294089:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756894.294095:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787756894.294104:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588654815488/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756894.294115:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756894.294117:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787756894.294119:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756894.294121:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9950 00000100:00080000:1.0:1787756894.294123:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756894.294125:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756894.294129:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787756894.294142:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654815616/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756894.294153:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756894.294157:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787756894.294173:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756894.294180:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9950 00000100:00100000:2.0:1787756894.294182:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654815360:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787756894.294185:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756894.294192:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756894.294197:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787756894.294209:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654815744/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756894.294217:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756894.294220:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787756894.294221:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756894.294224:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9950 00000100:00080000:1.0:1787756894.294228:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756894.294231:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756894.294234:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787756894.294244:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787756894.294245:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654815872/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756894.294253:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756894.294256:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787756894.294258:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756894.294261:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9950 00000100:00080000:1.0:1787756894.294266:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756894.294270:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756894.294274:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756894.294288:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756894/real 1787756894] req@ffff8c05b2d9b480 x1874588654815360/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787756894.294290:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654816000/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756894.294313:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787756894.294316:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654815360/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787756894.294320:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787756894.294325:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756894.294331:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787756894.294332:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9950 00000100:00080000:2.0:1787756894.294337:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787756894.294340:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787756894.294341:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654815360:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787756894.294350:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756894.294352:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654815488:192.168.201.107@tcp:38:kworker.0 00000008:00100000:1.0:1787756894.294359:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787756894.294376:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787756894.294377:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588654816128/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756894.294387:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756894/real 1787756894] req@ffff8c05b2d9b100 x1874588654815488/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787756894.294389:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756894.294399:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787756894.294403:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756894.294409:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9950 00000100:00100000:2.0:1787756894.294413:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654815488/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787756894.294414:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756894.294422:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756894.294430:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787756894.294443:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787756894.294446:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588654816256/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756894.294451:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756894.294460:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654815488:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756894.294470:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654815616:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756894.294490:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756894.294499:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756894/real 1787756894] req@ffff8c05b2d98380 x1874588654815616/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756894.294507:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654815616/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756894.294512:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756894.294514:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756894.294516:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654815616:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756894.294569:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654815744:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756894.294583:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756894.294591:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756894/real 1787756894] req@ffff8c05b2d99880 x1874588654815744/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756894.294599:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654815744/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756894.294604:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756894.294605:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756894.294608:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654815744:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756894.294611:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654815872:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756894.294621:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756894.294627:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756894/real 1787756894] req@ffff8c05b2d99f80 x1874588654815872/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756894.294634:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654815872/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756894.294639:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756894.294641:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756894.294643:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654815872:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756894.294646:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654816000:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756894.294655:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756894.294660:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756894/real 1787756894] req@ffff8c05b2d98000 x1874588654816000/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756894.294667:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654816000/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756894.294672:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756894.294673:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756894.294675:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654816000:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756894.294678:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654816128:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756894.294688:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756894.294693:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756894/real 1787756894] req@ffff8c05b2d98700 x1874588654816128/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756894.294701:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654816128/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756894.294706:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756894.294708:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756894.294709:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654816128:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756894.294713:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654816256:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756894.294722:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756894.294727:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756894/real 1787756894] req@ffff8c05b2d9a300 x1874588654816256/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756894.294734:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588654816256/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756894.294738:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756894.294740:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756894.294742:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654816256:192.168.201.107@tcp:8:kworker.0 00010000:00010000:3.0:1787756894.870065:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787756895.894254:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756896.918167:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756897.942309:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787756898.966218:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787756899.414052:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756899.414065:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787756899.414098:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756899.414114:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9955 00000100:00080000:1.0:1787756899.414127:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756899.414137:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787756899.414191:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588654816384/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756899.414245:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756899.414252:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787756899.414256:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756899.414264:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9955 00000100:00080000:1.0:1787756899.414273:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756899.414282:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756899.414312:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787756899.414337:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588654816512/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756899.414357:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756899.414366:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787756899.414373:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787756899.414375:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654816384:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787756899.414386:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9955 00000100:00080000:1.0:1787756899.414406:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756899.414415:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756899.414423:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787756899.414444:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588654816640/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756899.414457:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756899.414461:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787756899.414463:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756899.414467:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9955 00000100:00080000:1.0:1787756899.414470:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756899.414474:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756899.414479:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787756899.414489:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588654816768/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756899.414509:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756899.414513:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787756899.414515:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756899.414519:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9955 00000100:00080000:1.0:1787756899.414522:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756899.414527:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756899.414531:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787756899.414588:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654816896/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756899.414601:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756899.414607:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:2.0:1787756899.414610:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787756899.414612:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756899.414627:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9955 00000100:00080000:1.0:1787756899.414636:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756899.414642:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756899.414649:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787756899.414676:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654817024/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756899.414690:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756899.414694:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787756899.414696:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756899.414706:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9955 00000100:00080000:1.0:1787756899.414712:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756899.414717:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756899.414723:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787756899.414743:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654817152/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756899.414746:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756899/real 1787756899] req@ffff8c05b2d9aa00 x1874588654816384/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787756899.414784:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756899.414807:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787756899.414815:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756899.414833:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9955 00000100:00080000:1.0:1787756899.414838:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756899.414845:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756899.414851:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756899.414857:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588654816384/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787756899.414927:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654817280/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756899.414963:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756899.414976:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756899.414986:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654816384:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787756899.415039:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654816512:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756899.415110:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756899.415133:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756899/real 1787756899] req@ffff8c05b2d9bb80 x1874588654816512/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756899.415150:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588654816512/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756899.415161:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756899.415166:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756899.415170:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654816512:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756899.415179:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654816640:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756899.415204:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756899.415216:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756899/real 1787756899] req@ffff8c05b2d9a300 x1874588654816640/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756899.415231:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588654816640/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756899.415241:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756899.415244:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756899.415248:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654816640:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756899.415256:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654816768:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756899.415278:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756899.415289:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756899/real 1787756899] req@ffff8c05b2d98700 x1874588654816768/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756899.415303:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654816768/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756899.415313:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756899.415317:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756899.415320:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654816768:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756899.415327:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654816896:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756899.415347:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756899.415358:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756899/real 1787756899] req@ffff8c05b2d98000 x1874588654816896/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756899.415372:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654816896/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756899.415383:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756899.415386:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756899.415390:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654816896:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756899.415397:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654817024:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756899.415416:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756899.415427:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756899/real 1787756899] req@ffff8c05b2d99f80 x1874588654817024/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756899.415441:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654817024/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756899.415452:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756899.415455:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756899.415459:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654817024:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756899.415465:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654817152:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756899.415484:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756899.415495:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756899/real 1787756899] req@ffff8c05b2d99880 x1874588654817152/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756899.415509:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654817152/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756899.415518:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756899.415522:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756899.415525:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654817152:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756899.415532:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654817280:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756899.415550:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756899.415561:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756899/real 1787756899] req@ffff8c05b2d98380 x1874588654817280/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756899.415574:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654817280/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756899.415593:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756899.415597:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756899.415601:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654817280:192.168.201.107@tcp:8:kworker.0 00010000:00010000:1.0:1787756899.990303:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787756901.014177:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787756902.038248:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756903.062200:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787756904.086275:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756904.534039:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756904.534052:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787756904.534058:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756904.534070:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9960 00000100:00080000:3.0:1787756904.534097:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756904.534107:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787756904.534206:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588654817408/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756904.534244:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756904.534249:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756904.534263:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756904.534269:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9960 00000100:00080000:3.0:1787756904.534274:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756904.534278:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756904.534291:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756904.534328:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588654817536/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756904.534356:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787756904.534365:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654817408:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787756904.534365:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756904.534374:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756904.534387:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9960 00000100:00080000:3.0:1787756904.534391:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756904.534396:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756904.534403:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756904.534452:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588654817664/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756904.534473:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787756904.534477:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787756904.534489:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756904.534495:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756904.534506:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9960 00000100:00080000:3.0:1787756904.534515:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756904.534523:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756904.534534:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756904.534561:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588654817792/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756904.534575:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756904.534579:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1787756904.534588:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756904/real 1787756904] req@ffff8c05a9f78700 x1874588654817408/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756904.534589:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756904.534596:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9960 00000100:00080000:3.0:1787756904.534602:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756904.534616:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756904.534641:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756904.534657:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588654817920/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756904.534667:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756904.534672:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756904.534674:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756904.534680:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9960 00000100:00080000:3.0:1787756904.534684:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787756904.534692:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588654817408/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756904.534693:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756904.534704:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756904.534723:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588654818048/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756904.534730:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756904.534739:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756904.534745:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787756904.534748:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787756904.534755:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787756904.534758:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654817408:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787756904.534766:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9960 00000100:00080000:3.0:1787756904.534774:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756904.534781:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756904.534786:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756904.534796:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588654818176/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756904.534806:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756904.534811:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756904.534813:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756904.534818:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9960 00000100:00080000:3.0:1787756904.534822:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756904.534827:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756904.534836:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654817536:192.168.201.107@tcp:38:kworker.0 00000008:00100000:3.0:1787756904.534837:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756904.534854:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588654818304/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787756904.535031:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756904.535074:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756904/real 1787756904] req@ffff8c05a9f7a300 x1874588654817536/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756904.535094:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588654817536/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756904.535106:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756904.535110:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756904.535115:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654817536:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756904.535126:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654817664:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756904.535157:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756904.535171:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756904/real 1787756904] req@ffff8c05a9f7aa00 x1874588654817664/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756904.535186:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588654817664/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756904.535195:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756904.535200:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756904.535204:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654817664:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756904.535211:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654817792:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756904.535239:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756904.535259:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756904/real 1787756904] req@ffff8c05a9f79880 x1874588654817792/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756904.535283:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588654817792/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756904.535301:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756904.535308:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756904.535315:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654817792:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756904.535327:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654817920:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756904.535358:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756904.535373:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756904/real 1787756904] req@ffff8c05a9f78380 x1874588654817920/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756904.535389:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588654817920/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756904.535399:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756904.535403:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756904.535407:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654817920:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756904.535413:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654818048:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756904.535432:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756904.535443:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756904/real 1787756904] req@ffff8c05a9f7a680 x1874588654818048/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756904.535456:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588654818048/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756904.535466:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756904.535469:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756904.535473:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654818048:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756904.535479:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654818176:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756904.535497:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756904.535508:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756904/real 1787756904] req@ffff8c05a9f78a80 x1874588654818176/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756904.535522:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588654818176/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756904.535540:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756904.535544:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756904.535547:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654818176:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756904.535574:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654818304:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756904.535594:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756904.535605:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756904/real 1787756904] req@ffff8c05a9f7b480 x1874588654818304/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756904.535625:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588654818304/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756904.535641:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756904.535646:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756904.535653:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654818304:192.168.201.107@tcp:8:kworker.0 00010000:00010000:3.0:1787756905.110169:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756906.134143:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787756907.158123:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787756908.182311:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787756909.206097:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756909.654014:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756909.654027:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787756909.654032:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756909.654045:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9966 00000100:00080000:3.0:1787756909.654059:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756909.654067:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787756909.654114:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79c00 x1874588654818432/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756909.654167:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756909.654172:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756909.654175:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756909.654187:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9966 00000100:00080000:3.0:1787756909.654192:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756909.654197:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756909.654208:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756909.654227:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588654818560/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756909.654242:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756909.654247:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756909.654251:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756909.654255:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9966 00000100:00080000:3.0:1787756909.654260:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756909.654264:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756909.654272:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756909.654285:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654818432:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787756909.654292:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588654818688/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756909.654317:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756909.654323:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756909.654328:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756909.654335:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9966 00000100:00080000:3.0:1787756909.654345:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756909.654354:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756909.654364:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787756909.654391:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756909.654398:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588654818816/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756909.654422:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756909.654429:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756909.654434:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756909.654440:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9966 00000100:00080000:3.0:1787756909.654446:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756909.654451:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756909.654461:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756909.654463:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756909/real 1787756909] req@ffff8c05a9f79c00 x1874588654818432/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756909.654484:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588654818944/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756909.654498:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79c00 x1874588654818432/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756909.654507:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756909.654523:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756909.654531:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756909.654538:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756909.654543:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9966 00000100:00080000:2.0:1787756909.654551:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756909.654558:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654818432:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787756909.654566:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787756909.654575:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654818560:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787756909.654577:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756909.654592:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756909.654612:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588654819072/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756909.654634:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756909.654643:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1787756909.654645:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787756909.654647:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756909.654661:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9966 00000100:00080000:3.0:1787756909.654668:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756909.654674:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756909.654684:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756909.654694:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756909/real 1787756909] req@ffff8c05a9f7b480 x1874588654818560/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756909.654703:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588654819200/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756909.654756:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756909.654761:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756909.654764:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756909.654773:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9966 00000100:00100000:2.0:1787756909.654776:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588654818560/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756909.654778:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756909.654787:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756909.654797:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756909.654817:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588654819328/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756909.654818:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756909.654835:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756909.654850:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654818560:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756909.654969:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654818688:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756909.655005:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756909.655022:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756909/real 1787756909] req@ffff8c05a9f78a80 x1874588654818688/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756909.655038:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588654818688/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756909.655048:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756909.655053:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756909.655057:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654818688:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756909.655064:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654818816:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756909.655084:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756909.655095:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756909/real 1787756909] req@ffff8c05a9f7a680 x1874588654818816/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756909.655110:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588654818816/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756909.655120:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756909.655124:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756909.655127:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654818816:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756909.655134:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654818944:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756909.655154:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756909.655165:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756909/real 1787756909] req@ffff8c05a9f78380 x1874588654818944/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756909.655179:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588654818944/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756909.655188:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756909.655192:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756909.655195:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654818944:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756909.655201:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654819072:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756909.655232:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756909.655253:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756909/real 1787756909] req@ffff8c05a9f79880 x1874588654819072/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756909.655276:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588654819072/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756909.655305:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756909.655312:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756909.655318:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654819072:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756909.655331:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654819200:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756909.655374:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756909.655399:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756909/real 1787756909] req@ffff8c05a9f7aa00 x1874588654819200/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756909.655424:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588654819200/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756909.655441:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756909.655447:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756909.655453:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654819200:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756909.655466:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654819328:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756909.655496:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756909.655512:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756909/real 1787756909] req@ffff8c05a9f7a300 x1874588654819328/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756909.655527:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588654819328/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756909.655537:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756909.655541:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756909.655546:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654819328:192.168.201.107@tcp:8:kworker.0 00010000:00010000:1.0:1787756910.230209:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756911.254166:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787756912.278233:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756913.302102:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756914.326246:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756914.774106:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756914.774122:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787756914.774130:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756914.774149:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9971 00000100:00080000:3.0:1787756914.774169:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756914.774184:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787756914.774254:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588654819456/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756914.774311:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756914.774320:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756914.774325:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756914.774338:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9971 00000100:00080000:3.0:1787756914.774347:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756914.774359:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756914.774380:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756914.774416:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588654819584/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756914.774437:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756914.774445:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756914.774449:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756914.774456:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9971 00000100:00080000:3.0:1787756914.774464:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787756914.774470:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654819456:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787756914.774477:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756914.774499:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756914.774526:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588654819712/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756914.774562:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756914.774569:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756914.774573:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756914.774581:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9971 00000100:00080000:3.0:1787756914.774591:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756914.774601:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756914.774612:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787756914.774624:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756914.774647:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588654819840/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756914.774682:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756914.774690:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756914.774695:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756914.774705:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9971 00000100:00080000:3.0:1787756914.774714:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787756914.774722:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756914/real 1787756914] req@ffff8c05a9f78700 x1874588654819456/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756914.774731:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756914.774751:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756914.774780:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588654819456/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756914.774788:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588654819968/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756914.774844:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756914.774857:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756914.774873:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654819456:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787756914.774928:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756914.774936:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756914.774940:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756914.774953:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9971 00000100:00080000:3.0:1787756914.774967:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787756914.774976:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654819584:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787756914.774983:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756914.775001:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756914.775034:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588654820096/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787756914.775052:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787756914.775071:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756914.775081:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787756914.775086:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787756914.775096:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756914/real 1787756914] req@ffff8c05a9f78e00 x1874588654819584/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756914.775103:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9971 00000100:00080000:3.0:1787756914.775119:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756914.775134:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787756914.775144:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588654819584/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787756914.775152:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787756914.775175:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756914.775186:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756914.775196:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588654820224/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787756914.775203:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654819584:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787756914.775235:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756914.775242:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756914.775246:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756914.775289:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9971 00000100:00080000:3.0:1787756914.775298:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787756914.775301:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654819712:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787756914.775314:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756914.775329:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756914.775371:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588654820352/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787756914.775382:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756914.775414:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756914/real 1787756914] req@ffff8c05a9f79180 x1874588654819712/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756914.775442:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588654819712/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756914.775459:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756914.775467:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756914.775474:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654819712:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756914.775489:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654819840:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756914.775531:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756914.775554:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756914/real 1787756914] req@ffff8c05a9f79f80 x1874588654819840/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756914.775588:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588654819840/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756914.775603:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756914.775609:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756914.775614:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654819840:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756914.775625:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654819968:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756914.775663:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756914.775685:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756914/real 1787756914] req@ffff8c05a9f7ad80 x1874588654819968/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756914.775712:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588654819968/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756914.775733:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756914.775742:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756914.775749:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654819968:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756914.775766:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654820096:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756914.775843:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756914.775937:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756914/real 1787756914] req@ffff8c05a9f7bb80 x1874588654820096/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756914.775968:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588654820096/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756914.775999:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756914.776007:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756914.776016:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654820096:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756914.776045:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654820224:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756914.776094:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756914.776126:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756914/real 1787756914] req@ffff8c05a9f7b800 x1874588654820224/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756914.776150:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588654820224/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756914.776177:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756914.776184:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756914.776191:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654820224:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756914.776336:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654820352:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756914.776405:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756914.776446:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756914/real 1787756914] req@ffff8c05a9f7b100 x1874588654820352/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756914.776473:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588654820352/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756914.776490:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756914.776499:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756914.776508:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654820352:192.168.201.107@tcp:8:kworker.0 00010000:00010000:1.0:1787756915.350104:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787756916.374267:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787756917.398207:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787756918.422248:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756919.446162:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787756919.894007:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756919.894020:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787756919.894026:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756919.894039:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9976 00000100:00080000:0.0:1787756919.894051:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756919.894060:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787756919.894109:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588654820480/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756919.894152:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756919.894157:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787756919.894159:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756919.894165:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9976 00000100:00080000:0.0:1787756919.894169:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756919.894174:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756919.894185:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756919.894200:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588654820608/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756919.894213:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756919.894218:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787756919.894220:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756919.894224:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9976 00000100:00080000:0.0:1787756919.894227:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756919.894232:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756919.894237:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756919.894248:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588654820736/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756919.894260:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756919.894265:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787756919.894268:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756919.894272:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9976 00000100:00080000:0.0:1787756919.894276:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756919.894281:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756919.894286:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787756919.894297:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588654820864/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756919.894311:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787756919.894314:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654820480:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787756919.894318:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787756919.894326:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756919.894334:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9976 00000100:00080000:0.0:1787756919.894338:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756919.894343:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756919.894350:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756919.894364:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588654820992/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756919.894377:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756919.894382:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787756919.894385:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756919.894389:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9976 00000100:00080000:0.0:1787756919.894398:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756919.894402:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756919.894409:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787756919.894417:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756919.894430:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588654821120/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756919.894445:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756919.894450:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787756919.894452:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756919.894457:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9976 00000100:00080000:0.0:1787756919.894462:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756919.894466:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756919.894472:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787756919.894488:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588654821248/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756919.894499:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756919.894503:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787756919.894505:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756919.894515:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9976 00000100:00100000:2.0:1787756919.894515:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756919/real 1787756919] req@ffff8c05a9f77100 x1874588654820480/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756919.894525:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756919.894537:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756919.894552:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787756919.894570:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588654821376/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756919.894594:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588654820480/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756919.894614:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756919.894621:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756919.894626:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654820480:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787756919.894638:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654820608:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756919.894673:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756919.894690:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756919/real 1787756919] req@ffff8c05a9f74000 x1874588654820608/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756919.894749:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588654820608/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756919.894760:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756919.894764:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756919.894779:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654820608:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756919.894787:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654820736:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756919.894815:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756919.894833:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756919/real 1787756919] req@ffff8c05a9f75880 x1874588654820736/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756919.894856:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588654820736/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756919.894873:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756919.894936:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756919.894944:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654820736:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756919.894980:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654820864:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756919.895013:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756919.895029:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756919/real 1787756919] req@ffff8c05a9f75500 x1874588654820864/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756919.895044:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588654820864/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756919.895055:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756919.895058:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756919.895062:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654820864:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756919.895069:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654820992:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756919.895088:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756919.895106:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756919/real 1787756919] req@ffff8c05a9f74e00 x1874588654820992/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756919.895119:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588654820992/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756919.895130:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756919.895134:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756919.895137:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654820992:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756919.895143:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654821120:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756919.895160:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756919.895171:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756919/real 1787756919] req@ffff8c05a9f76300 x1874588654821120/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756919.895184:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588654821120/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756919.895194:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756919.895197:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756919.895201:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654821120:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756919.895207:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654821248:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756919.895224:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756919.895234:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756919/real 1787756919] req@ffff8c05a9f77800 x1874588654821248/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756919.895247:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588654821248/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756919.895257:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756919.895260:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756919.895265:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654821248:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756919.895271:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654821376:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756919.895289:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756919.895299:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756919/real 1787756919] req@ffff8c05a9f75180 x1874588654821376/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756919.895322:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588654821376/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756919.895332:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756919.895335:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756919.895338:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654821376:192.168.201.107@tcp:8:kworker.0 00010000:00010000:1.0:1787756920.470156:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756921.494153:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756922.518158:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787756923.542161:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787756924.566277:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787756925.014013:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756925.014031:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787756925.014038:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756925.014054:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9981 00000100:00080000:0.0:1787756925.014081:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756925.014093:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787756925.014147:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588654821504/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756925.014186:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756925.014191:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787756925.014194:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756925.014200:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9981 00000100:00080000:0.0:1787756925.014205:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756925.014210:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756925.014220:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756925.014237:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588654821632/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756925.014250:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756925.014254:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787756925.014257:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756925.014261:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9981 00000100:00080000:0.0:1787756925.014266:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756925.014271:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756925.014278:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756925.014290:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588654821760/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756925.014304:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756925.014309:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787756925.014311:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787756925.014321:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654821504:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787756925.014322:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9981 00000100:00080000:0.0:1787756925.014332:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756925.014340:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756925.014347:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787756925.014362:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588654821888/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756925.014375:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756925.014380:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787756925.014383:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756925.014387:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9981 00000100:00080000:0.0:1787756925.014391:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756925.014396:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756925.014402:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756925.014417:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588654822016/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787756925.014424:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787756925.014451:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756925.014456:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787756925.014458:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756925.014464:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9981 00000100:00080000:0.0:1787756925.014468:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756925.014474:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756925.014481:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756925.014491:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756925/real 1787756925] req@ffff8c05a9f75180 x1874588654821504/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756925.014499:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588654822144/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756925.014542:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756925.014547:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787756925.014549:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756925.014556:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9981 00000100:00080000:0.0:1787756925.014559:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756925.014563:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756925.014567:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756925.014575:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588654821504/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756925.014583:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588654822272/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756925.014654:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756925.014658:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756925.014673:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787756925.014677:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787756925.014688:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787756925.014694:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654821504:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787756925.014702:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9981 00000100:00080000:0.0:1787756925.014716:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787756925.014724:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654821632:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787756925.014732:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756925.014749:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787756925.014780:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588654822400/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787756925.014795:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756925.014837:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756925/real 1787756925] req@ffff8c05a9f77800 x1874588654821632/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756925.014868:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588654821632/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756925.014949:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756925.014954:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756925.014978:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654821632:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756925.015010:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654821760:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756925.015043:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756925.015060:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756925/real 1787756925] req@ffff8c05a9f76300 x1874588654821760/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756925.015076:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588654821760/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756925.015087:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756925.015091:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756925.015095:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654821760:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756925.015109:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654821888:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756925.015128:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756925.015140:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756925/real 1787756925] req@ffff8c05a9f74e00 x1874588654821888/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756925.015154:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588654821888/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756925.015164:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756925.015168:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756925.015172:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654821888:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756925.015178:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654822016:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756925.015195:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756925.015206:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756925/real 1787756925] req@ffff8c05a9f75500 x1874588654822016/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756925.015220:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588654822016/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756925.015230:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756925.015235:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756925.015243:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654822016:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756925.015252:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654822144:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756925.015275:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756925.015286:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756925/real 1787756925] req@ffff8c05a9f75880 x1874588654822144/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756925.015323:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588654822144/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756925.015339:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756925.015343:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756925.015347:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654822144:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756925.015355:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654822272:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756925.015377:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756925.015389:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756925/real 1787756925] req@ffff8c05a9f74000 x1874588654822272/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756925.015403:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588654822272/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756925.015413:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756925.015416:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756925.015420:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654822272:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756925.015426:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654822400:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756925.015444:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756925.015454:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756925/real 1787756925] req@ffff8c05a9f77100 x1874588654822400/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756925.015468:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588654822400/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756925.015477:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756925.015480:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756925.015484:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654822400:192.168.201.107@tcp:8:kworker.0 00010000:00010000:1.0:1787756925.590113:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787756926.614289:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756927.638110:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787756928.662227:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756929.686156:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756930.134089:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756930.134106:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787756930.134114:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756930.134130:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9986 00000100:00080000:3.0:1787756930.134148:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756930.134160:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787756930.134227:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588654822528/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756930.134270:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756930.134275:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756930.134277:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756930.134283:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9986 00000100:00080000:3.0:1787756930.134287:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756930.134292:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756930.134306:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756930.134326:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588654822656/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756930.134339:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756930.134344:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756930.134347:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756930.134352:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9986 00000100:00080000:3.0:1787756930.134356:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756930.134360:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756930.134366:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756930.134378:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588654822784/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756930.134390:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756930.134395:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756930.134398:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756930.134408:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9986 00000100:00080000:3.0:1787756930.134413:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756930.134418:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756930.134425:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756930.134435:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588654822912/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756930.134449:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756930.134453:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756930.134458:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787756930.134471:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654822528:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787756930.134473:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9986 00000100:00080000:3.0:1787756930.134494:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756930.134507:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756930.134519:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756930.134575:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588654823040/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756930.134597:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756930.134604:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756930.134609:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756930.134619:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9986 00000100:00080000:3.0:1787756930.134630:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756930.134641:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:2.0:1787756930.134648:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:3.0:1787756930.134654:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756930.134675:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588654823168/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756930.134689:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756930.134694:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787756930.134696:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756930.134702:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9986 00000100:00080000:3.0:1787756930.134709:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756930.134716:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756930.134723:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756930.134736:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588654823296/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756930.134750:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756930.134755:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756930.134758:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756930.134772:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9986 00000100:00100000:2.0:1787756930.134773:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756930/real 1787756930] req@ffff8c05a9f79500 x1874588654822528/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756930.134784:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756930.134797:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756930.134808:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756930.134827:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588654823424/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756930.134851:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588654822528/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756930.134959:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756930.134971:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756930.134980:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654822528:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787756930.134997:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654822656:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756930.135048:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756930.135076:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756930/real 1787756930] req@ffff8c05a9f7b100 x1874588654822656/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756930.135104:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588654822656/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756930.135122:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756930.135130:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756930.135138:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654822656:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756930.135151:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654822784:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756930.135191:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756930.135212:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756930/real 1787756930] req@ffff8c05a9f7b800 x1874588654822784/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756930.135239:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588654822784/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756930.135268:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756930.135276:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756930.135283:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654822784:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756930.135329:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654822912:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756930.135373:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756930.135397:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756930/real 1787756930] req@ffff8c05a9f7bb80 x1874588654822912/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756930.135424:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588654822912/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756930.135443:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756930.135450:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756930.135458:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654822912:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756930.135472:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654823040:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756930.135509:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756930.135562:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756930/real 1787756930] req@ffff8c05a9f7ad80 x1874588654823040/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756930.135590:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588654823040/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756930.135608:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756930.135616:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756930.135624:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654823040:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756930.135636:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654823168:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756930.135677:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756930.135701:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756930/real 1787756930] req@ffff8c05a9f79f80 x1874588654823168/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756930.135726:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588654823168/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756930.135745:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756930.135752:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756930.135759:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654823168:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756930.135773:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654823296:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756930.135811:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756930.135834:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756930/real 1787756930] req@ffff8c05a9f79180 x1874588654823296/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756930.135858:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588654823296/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756930.135915:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756930.135924:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756930.135932:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654823296:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756930.135946:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654823424:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756930.135986:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756930.136011:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756930/real 1787756930] req@ffff8c05a9f78e00 x1874588654823424/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756930.136036:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588654823424/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756930.136055:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756930.136062:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756930.136069:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654823424:192.168.201.107@tcp:8:kworker.0 00010000:00010000:3.0:1787756930.710281:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787756931.734065:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787756932.758258:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787756933.782067:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787756934.806175:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756935.253960:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756935.253972:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787756935.253977:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756935.253989:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9991 00000100:00080000:3.0:1787756935.254001:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756935.254010:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787756935.254071:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588654823552/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756935.254122:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756935.254129:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756935.254156:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756935.254165:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9991 00000100:00080000:3.0:1787756935.254174:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756935.254185:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756935.254202:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756935.254224:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654823552:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787756935.254240:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588654823680/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756935.254282:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756935.254290:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756935.254295:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756935.254304:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9991 00000100:00080000:3.0:1787756935.254314:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756935.254322:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756935.254331:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756935.254353:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588654823808/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756935.254369:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756935.254377:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787756935.254378:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654823680:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787756935.254381:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756935.254398:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9991 00000100:00080000:3.0:1787756935.254405:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756935.254412:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756935.254419:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756935.254477:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654823808:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756935.254490:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588654823936/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756935.254513:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756935.254520:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756935.254526:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787756935.254527:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654823936:192.168.201.107@tcp:8:kworker.0 00000100:00080000:3.0:1787756935.254543:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9991 00000100:00080000:3.0:1787756935.254570:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756935.254579:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756935.254589:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756935.254610:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588654824064/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756935.254643:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756935.254647:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756935.254650:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756935.254654:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9991 00000100:00080000:3.0:1787756935.254659:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756935.254664:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756935.254669:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756935.254692:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588654824192/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756935.254698:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654824064:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787756935.254725:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756935.254730:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787756935.254733:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756935.254747:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9991 00000100:00080000:3.0:1787756935.254754:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787756935.254758:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654824192:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787756935.254768:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756935.254781:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756935.254813:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588654824320/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756935.254861:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756935.254867:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756935.254871:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787756935.254914:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654824320:192.168.201.107@tcp:8:kworker.0 00000100:00080000:3.0:1787756935.254918:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9991 00000100:00080000:3.0:1787756935.254930:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756935.254938:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756935.254950:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756935.254978:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588654824448/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756935.255029:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654824448:192.168.201.107@tcp:8:kworker.0 00010000:00010000:3.0:1787756935.830104:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787756936.854205:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756937.878191:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787756938.326430:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.107:988 00000400:00000100:1.0:1787756938.326506:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.107@tcp at host 192.168.201.107 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787756938.326534:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787756938.326614:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787756938.326648:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787756938.326675:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787756938.326704:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787756938.326723:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787756938.326744:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:2.0:1787756938.326763:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756935/real 1787756938] req@ffff8c05a9f79880 x1874588654823936/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787756938.326782:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:2.0:1787756938.326854:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588654823936/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756938.326986:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756938.326998:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756938.327008:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654823936:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756938.327027:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756935/real 1787756938] req@ffff8c05a9f7a300 x1874588654823680/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756938.327057:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588654823680/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756938.327072:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756938.327079:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756938.327086:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654823680:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756938.327099:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756935/real 1787756938] req@ffff8c05a9f7aa00 x1874588654823808/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756938.327137:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588654823808/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756938.327154:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756938.327161:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756938.327168:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654823808:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756938.327183:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756935/real 1787756938] req@ffff8c05a9f78700 x1874588654823552/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756938.327208:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588654823552/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756938.327248:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756938.327255:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756938.327263:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654823552:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787756938.327334:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756935/real 1787756938] req@ffff8c05a9f7b480 x1874588654824448/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756938.327360:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588654824448/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756938.327375:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756938.327414:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756938.327422:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654824448:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756938.327436:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756935/real 1787756938] req@ffff8c05a9f78a80 x1874588654824320/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756938.327458:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588654824320/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756938.327475:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756938.327481:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756938.327488:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654824320:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756938.327500:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756935/real 1787756938] req@ffff8c05a9f7a680 x1874588654824192/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756938.327524:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588654824192/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756938.327541:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756938.327547:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756938.327554:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654824192:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756938.327568:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756935/real 1787756938] req@ffff8c05a9f78380 x1874588654824064/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756938.327591:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588654824064/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756938.327605:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756938.327611:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756938.327618:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654824064:192.168.201.107@tcp:38:kworker.0 00010000:00010000:3.0:1787756938.902236:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787756939.926128:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787756940.374070:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756940.374088:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787756940.374095:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756940.374111:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 9996 00000100:00080000:0.0:1787756940.374128:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756940.374139:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787756940.374185:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588654824576/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756940.374226:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756940.374231:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787756940.374233:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756940.374239:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9996 00000100:00080000:0.0:1787756940.374244:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756940.374249:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756940.374260:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756940.374277:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588654824704/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756940.374293:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756940.374301:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787756940.374308:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787756940.374310:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654824576:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787756940.374317:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9996 00000100:00080000:0.0:1787756940.374328:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756940.374332:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756940.374378:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756940.374400:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588654824832/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756940.374415:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756940.374420:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787756940.374423:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756940.374428:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 9996 00000100:00080000:0.0:1787756940.374444:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787756940.374457:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787756940.374458:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756940.374489:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787756940.374520:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588654824960/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756940.374531:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756940/real 1787756940] req@ffff8c05a9f74a80 x1874588654824576/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756940.374582:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756940.374590:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787756940.374594:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756940.374607:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9996 00000100:00100000:2.0:1787756940.374608:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588654824576/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756940.374620:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756940.374640:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:2.0:1787756940.374646:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787756940.374655:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787756940.374663:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756940.374675:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654824576:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787756940.374693:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588654825088/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756940.374721:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787756940.374723:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654824704:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787756940.374737:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787756940.374745:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756940.374756:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9996 00000100:00080000:0.0:1787756940.374768:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756940.374778:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756940.374790:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787756940.374794:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756940.374827:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588654825216/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756940.374848:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756940/real 1787756940] req@ffff8c05a9f76680 x1874588654824704/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756940.374854:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756940.374865:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787756940.374869:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756940.374946:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9996 00000100:00080000:0.0:1787756940.374955:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756940.374988:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756940.375000:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756940.375027:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588654824704/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756940.375038:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588654825344/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756940.375066:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756940.375082:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756940.375084:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787756940.375102:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787756940.375103:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654824704:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787756940.375114:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787756940.375127:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654824832:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787756940.375131:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 9996 00000100:00080000:0.0:1787756940.375148:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756940.375160:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787756940.375173:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787756940.375201:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756940.375216:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588654825472/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787756940.375237:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756940/real 1787756940] req@ffff8c05a9f77b80 x1874588654824832/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756940.375270:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588654824832/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756940.375288:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756940.375297:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756940.375304:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654824832:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756940.375319:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654824960:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756940.375394:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756940.375422:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756940/real 1787756940] req@ffff8c05a9f76d80 x1874588654824960/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756940.375462:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588654824960/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756940.375480:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756940.375487:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756940.375495:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654824960:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756940.375532:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654825088:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756940.375578:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756940.375608:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756940/real 1787756940] req@ffff8c05a9f76a00 x1874588654825088/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756940.375632:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588654825088/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756940.375649:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756940.375654:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756940.375660:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654825088:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756940.375697:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654825216:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787756940.375739:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756940.375762:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756940/real 1787756940] req@ffff8c05a9f74380 x1874588654825216/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756940.375785:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588654825216/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756940.375813:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756940.375820:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756940.375826:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654825216:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756940.375837:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654825344:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756940.375917:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756940.375944:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756940/real 1787756940] req@ffff8c05a9f75c00 x1874588654825344/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756940.375967:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588654825344/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756940.375983:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756940.375989:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756940.375995:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654825344:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756940.376007:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654825472:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787756940.376045:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756940.376067:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756940/real 1787756940] req@ffff8c05a9f74a80 x1874588654825472/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756940.376088:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588654825472/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756940.376102:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756940.376108:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756940.376115:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654825472:192.168.201.107@tcp:8:kworker.0 00010000:00010000:3.0:1787756940.905608:0:233476:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0000-mdc-ffff8c0583595800 00010000:00010000:3.0:1787756940.905725:0:233476:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0001-mdc-ffff8c0583595800 00010000:00010000:1.0:1787756940.905735:0:233477:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0000-mdc-ffff8c0589b26800 00010000:00010000:3.0:1787756940.905793:0:233476:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0001-mdc-ffff8c0589b26800 00010000:00010000:1.0:1787756940.950179:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787756941.974145:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787756942.998186:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787756944.022127:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1787756944.615182:0:233583:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8c059191c2c8),name=/ 00000002:00010000:3.0:1787756944.615217:0:233583:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787756944.615340:0:233583:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8c0589b26800 lock: ffff8c0591285000/0x109d23f2ff214c30 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 233583 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787756944.615361:0:233583:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8c0589b26800 lock: ffff8c0591285000/0x109d23f2ff214c30 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 233583 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787756944.615381:0:233583:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8c0589b26800 lock: ffff8c0591285000/0x109d23f2ff214c30 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 233583 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00080000:3.0:1787756944.615434:0:233583:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff8c05a9f79c00 x1874588654825600/t0(0) o101->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 576/1152 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1787756944.615449:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787756945.046249:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756945.494043:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756945.494061:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787756945.494067:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756945.494080:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10001 00000100:00080000:2.0:1787756945.494095:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756945.494105:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756945.494156:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73800 x1874588654825728/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756945.494233:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756945.494251:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756945.494282:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756945.494293:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10001 00000100:00080000:2.0:1787756945.494302:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756945.494311:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787756945.494325:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654825728:192.168.201.107@tcp:250:kworker.0 00000008:00100000:2.0:1787756945.494329:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756945.494377:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588654825856/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787756945.494421:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787756945.494422:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756945.494432:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756945.494436:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756945.494444:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10001 00000100:00080000:2.0:1787756945.494451:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756945.494459:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756945.494469:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756945.494491:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588654825984/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787756945.494497:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756945/real 1787756945] req@ffff8c05b2e73800 x1874588654825728/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756945.494546:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756945.494560:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756945.494563:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756945.494571:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10001 00000100:00100000:3.0:1787756945.494576:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73800 x1874588654825728/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756945.494582:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756945.494597:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756945.494612:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787756945.494638:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787756945.494639:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588654826112/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756945.494656:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756945.494675:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654825728:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787756945.494700:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756945.494705:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756945.494708:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756945.494714:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10001 00000100:00080000:2.0:1787756945.494720:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787756945.494728:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654825856:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787756945.494729:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756945.494746:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756945.494768:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588654826240/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756945.494786:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787756945.494787:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787756945.494799:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756945.494803:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756945.494811:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10001 00000100:00080000:2.0:1787756945.494817:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787756945.494825:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756945/real 1787756945] req@ffff8c05b2e70380 x1874588654825856/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756945.494826:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756945.494842:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756945.494869:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588654826368/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787756945.494875:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588654825856/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756945.494956:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756945.494961:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756945.494971:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787756945.494977:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787756945.494981:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787756945.494992:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654825856:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787756945.495001:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10001 00000100:00080000:2.0:1787756945.495021:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787756945.495023:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654825984:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787756945.495041:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756945.495057:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787756945.495079:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756945.495087:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588654826496/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756945.495106:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787756945.495107:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756945/real 1787756945] req@ffff8c05b2e71880 x1874588654825984/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756945.495124:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756945.495132:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756945.495150:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10001 00000100:00080000:2.0:1787756945.495157:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787756945.495160:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588654825984/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756945.495168:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756945.495186:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787756945.495190:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756945.495199:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756945.495211:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588654826624/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787756945.495213:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654825984:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756945.495240:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654826112:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787756945.495305:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756945.495331:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756945/real 1787756945] req@ffff8c05b2e71c00 x1874588654826112/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756945.495348:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588654826112/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756945.495364:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756945.495369:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756945.495374:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654826112:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787756945.495397:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654826240:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787756945.495428:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756945.495444:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756945/real 1787756945] req@ffff8c05b2e73b80 x1874588654826240/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756945.495463:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588654826240/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756945.495477:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756945.495482:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756945.495489:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654826240:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756945.495498:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654826368:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787756945.495530:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756945.495548:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756945/real 1787756945] req@ffff8c05b2e71500 x1874588654826368/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756945.495572:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588654826368/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756945.495583:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756945.495587:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756945.495591:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654826368:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787756945.495602:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654826496:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787756945.495633:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756945.495649:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756945/real 1787756945] req@ffff8c05b2e73480 x1874588654826496/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756945.495665:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588654826496/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756945.495687:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756945.495692:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756945.495696:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654826496:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787756945.495705:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654826624:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787756945.495731:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756945.495753:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756945/real 1787756945] req@ffff8c05b2e70700 x1874588654826624/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756945.495769:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588654826624/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756945.495783:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787756945.495788:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787756945.495795:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654826624:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787756945.622047:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787756946.070144:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787756946.646114:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787756947.094182:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787756947.670038:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787756948.118159:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787756948.693970:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787756949.142223:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787756949.717994:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787756950.166075:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756950.614035:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756950.614050:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787756950.614055:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756950.614070:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10007 00000100:00080000:2.0:1787756950.614085:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756950.614096:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756950.614153:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588654826752/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756950.614230:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756950.614236:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756950.614240:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756950.614248:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10007 00000100:00080000:2.0:1787756950.614258:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756950.614270:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756950.614287:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756950.614301:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654826752:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787756950.614324:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588654826880/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756950.614353:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756950.614361:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756950.614365:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756950.614371:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10007 00000100:00080000:2.0:1787756950.614380:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756950.614390:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:0.0:1787756950.614392:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:2.0:1787756950.614407:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756950.614429:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588654827008/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756950.614445:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787756950.614454:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756950/real 1787756950] req@ffff8c05b2e70000 x1874588654826752/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756950.614454:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756950.614462:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756950.614478:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10007 00000100:00080000:2.0:1787756950.614488:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756950.614494:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756950.614501:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787756950.614503:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588654826752/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756950.614526:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787756950.614529:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588654827136/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756950.614539:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756950.614556:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654826752:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787756950.614557:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756950.614575:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756950.614579:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756950.614589:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10007 00000100:00080000:2.0:1787756950.614598:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787756950.614604:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654826880:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787756950.614615:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756950.614628:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787756950.614643:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756950.614662:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588654827264/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787756950.614670:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756950/real 1787756950] req@ffff8c05b2e73100 x1874588654826880/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756950.614724:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588654826880/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756950.614730:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756950.614745:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756950.614753:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756950.614772:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756950.614773:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10007 00000100:00080000:2.0:1787756950.614788:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756950.614795:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756950.614806:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654826880:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787756950.614817:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756950.614829:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756950.614833:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654827008:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756950.614855:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588654827392/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756950.614873:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756950.614927:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787756950.614932:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787756950.614944:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787756950.614952:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10007 00000100:00080000:2.0:1787756950.614965:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787756950.614975:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756950/real 1787756950] req@ffff8c05b2e72d80 x1874588654827008/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756950.614978:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756950.614996:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787756950.615012:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588654827008/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756950.615026:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588654827520/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756950.615032:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756950.615050:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787756950.615062:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787756950.615063:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654827008:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787756950.615079:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756950.615083:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787756950.615088:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654827136:192.168.201.107@tcp:8:kworker.0 00000100:00080000:2.0:1787756950.615095:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10007 00000100:00080000:2.0:1787756950.615107:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756950.615114:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756950.615124:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787756950.615147:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756950.615155:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588654827648/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787756950.615210:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756950/real 1787756950] req@ffff8c05b2e72680 x1874588654827136/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756950.615239:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588654827136/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756950.615251:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756950.615257:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756950.615262:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654827136:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756950.615286:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654827264:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756950.615315:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756950.615329:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756950/real 1787756950] req@ffff8c05b2e70e00 x1874588654827264/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756950.615343:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588654827264/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756950.615354:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756950.615357:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756950.615361:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654827264:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756950.615368:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654827392:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756950.615386:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756950.615397:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756950/real 1787756950] req@ffff8c05b2e72a00 x1874588654827392/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756950.615411:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588654827392/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756950.615420:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756950.615424:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756950.615427:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654827392:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756950.615433:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654827520:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756950.615451:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756950.615462:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756950/real 1787756950] req@ffff8c05b2e71180 x1874588654827520/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756950.615476:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588654827520/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756950.615485:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756950.615489:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756950.615492:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654827520:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756950.615498:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654827648:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756950.615516:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756950.615526:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756950/real 1787756950] req@ffff8c05b2e72300 x1874588654827648/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756950.615540:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588654827648/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756950.615549:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756950.615553:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756950.615556:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654827648:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787756950.742022:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787756951.190264:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787756951.765991:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787756952.214100:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787756952.790000:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787756953.238283:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787756953.814001:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787756954.262187:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787756954.838010:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787756955.286290:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787756955.734055:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756955.734072:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787756955.734101:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756955.734117:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10012 00000100:00080000:1.0:1787756955.734135:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756955.734147:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787756955.734197:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588654827776/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756955.734239:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756955.734244:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787756955.734247:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756955.734252:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10012 00000100:00080000:1.0:1787756955.734257:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756955.734261:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756955.734273:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787756955.734289:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588654827904/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756955.734307:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756955.734314:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787756955.734318:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756955.734327:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10012 00000100:00080000:1.0:1787756955.734333:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787756955.734339:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654827776:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787756955.734348:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756955.734368:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787756955.734400:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654828032/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756955.734421:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756955.734426:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787756955.734428:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756955.734435:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10012 00000100:00080000:1.0:1787756955.734441:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787756955.734455:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787756955.734462:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756955.734475:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787756955.734496:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654828160/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756955.734509:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756955.734514:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787756955.734517:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756955.734527:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10012 00000100:00100000:0.0:1787756955.734533:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756955/real 1787756955] req@ffff8c05b2d9b100 x1874588654827776/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787756955.734534:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756955.734547:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756955.734558:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787756955.734579:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654828288/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756955.734593:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787756955.734597:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654827776/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787756955.734599:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787756955.734606:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756955.734619:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10012 00000100:00080000:1.0:1787756955.734629:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756955.734639:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756955.734649:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787756955.734651:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787756955.734663:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654828416/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756955.734667:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756955.734687:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654827776:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787756955.734690:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756955.734704:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787756955.734707:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756955.734715:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10012 00000100:00100000:0.0:1787756955.734721:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654827904:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787756955.734722:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756955.734735:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756955.734745:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787756955.734764:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588654828544/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756955.734776:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756955.734782:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787756955.734787:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787756955.734798:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787756955.734804:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10012 00000100:00080000:1.0:1787756955.734818:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756955.734827:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787756955.734839:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756955/real 1787756955] req@ffff8c05b2d9b480 x1874588654827904/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787756955.734842:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787756955.734934:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588654828672/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787756955.734990:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654827904/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756955.735014:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756955.735022:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756955.735030:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654827904:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756955.735071:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654828032:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756955.735143:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756955.735167:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756955/real 1787756955] req@ffff8c05b2d98380 x1874588654828032/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756955.735193:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654828032/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756955.735212:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756955.735220:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756955.735227:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654828032:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756955.735241:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654828160:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756955.735280:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756955.735302:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756955/real 1787756955] req@ffff8c05b2d99880 x1874588654828160/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756955.735327:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654828160/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756955.735345:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756955.735352:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756955.735360:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654828160:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756955.735372:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654828288:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756955.735407:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756955.735427:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756955/real 1787756955] req@ffff8c05b2d99f80 x1874588654828288/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756955.735453:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654828288/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756955.735472:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756955.735479:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756955.735487:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654828288:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756955.735499:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654828416:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756955.735536:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756955.735559:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756955/real 1787756955] req@ffff8c05b2d98000 x1874588654828416/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756955.735584:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654828416/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756955.735603:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756955.735609:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756955.735615:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654828416:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756955.735628:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654828544:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756955.735665:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756955.735686:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756955/real 1787756955] req@ffff8c05b2d98700 x1874588654828544/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756955.735712:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654828544/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756955.735730:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756955.735736:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756955.735743:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654828544:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756955.735754:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654828672:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756955.735792:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756955.735813:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756955/real 1787756955] req@ffff8c05b2d9a300 x1874588654828672/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756955.735828:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588654828672/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756955.735847:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756955.735851:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756955.735855:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654828672:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787756955.862000:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787756956.310135:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787756956.886020:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787756957.334237:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787756957.909995:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787756958.358138:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787756958.933979:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787756959.382263:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787756959.957984:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787756960.406138:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787756960.854022:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756960.854041:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787756960.854048:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756960.854064:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10017 00000100:00080000:1.0:1787756960.854091:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756960.854101:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787756960.854150:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588654828800/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756960.854187:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756960.854192:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787756960.854195:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756960.854200:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10017 00000100:00080000:1.0:1787756960.854204:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756960.854208:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756960.854220:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787756960.854238:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588654828928/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756960.854252:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756960.854258:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787756960.854260:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756960.854266:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10017 00000100:00080000:1.0:1787756960.854270:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756960.854274:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756960.854281:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787756960.854298:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588654829056/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787756960.854305:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654828800:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787756960.854314:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756960.854325:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787756960.854328:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756960.854343:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10017 00000100:00080000:1.0:1787756960.854347:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756960.854352:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756960.854360:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787756960.854378:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588654829184/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756960.854391:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756960.854396:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787756960.854399:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756960.854403:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10017 00000100:00080000:1.0:1787756960.854409:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756960.854413:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756960.854421:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787756960.854425:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787756960.854441:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654829312/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756960.854456:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756960.854460:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787756960.854463:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756960.854467:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10017 00000100:00080000:1.0:1787756960.854473:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756960.854479:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787756960.854493:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756960/real 1787756960] req@ffff8c05b2d9bb80 x1874588654828800/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787756960.854495:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787756960.854522:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654829440/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787756960.854534:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588654828800/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787756960.854543:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756960.854559:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787756960.854565:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787756960.854574:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787756960.854576:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10017 00000100:00080000:1.0:1787756960.854588:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756960.854592:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787756960.854599:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756960.854606:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787756960.854618:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654828800:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787756960.854626:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654829568/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787756960.854642:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654828928:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787756960.854645:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756960.854657:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787756960.854661:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756960.854668:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10017 00000100:00080000:1.0:1787756960.854674:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756960.854679:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756960.854686:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787756960.854694:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787756960.854707:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654829696/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787756960.854720:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756960/real 1787756960] req@ffff8c05b2d9aa00 x1874588654828928/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756960.854743:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588654828928/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756960.854758:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756960.854768:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756960.854777:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654828928:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756960.854821:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654829056:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756960.854865:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756960.854931:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756960/real 1787756960] req@ffff8c05b2d9a300 x1874588654829056/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756960.854948:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588654829056/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756960.854958:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756960.854963:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756960.854967:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654829056:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756960.854975:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654829184:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756960.855046:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756960.855063:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756960/real 1787756960] req@ffff8c05b2d98700 x1874588654829184/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756960.855078:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654829184/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756960.855088:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756960.855092:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756960.855096:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654829184:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756960.855103:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654829312:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756960.855125:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756960.855136:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756960/real 1787756960] req@ffff8c05b2d98000 x1874588654829312/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756960.855151:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654829312/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756960.855161:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756960.855164:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756960.855168:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654829312:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756960.855175:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654829440:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756960.855194:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756960.855205:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756960/real 1787756960] req@ffff8c05b2d99f80 x1874588654829440/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756960.855227:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654829440/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756960.855245:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756960.855251:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756960.855257:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654829440:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756960.855268:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654829568:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756960.855305:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756960.855326:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756960/real 1787756960] req@ffff8c05b2d99880 x1874588654829568/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756960.855342:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654829568/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756960.855361:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756960.855365:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756960.855369:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654829568:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756960.855376:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654829696:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756960.855400:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756960.855412:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756960/real 1787756960] req@ffff8c05b2d98380 x1874588654829696/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756960.855425:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654829696/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756960.855435:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756960.855438:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756960.855442:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654829696:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787756960.982050:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787756961.430293:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787756962.006004:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787756962.454160:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787756963.030075:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787756963.478160:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787756964.054043:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787756964.502114:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787756965.078062:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787756965.526212:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756965.974070:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756965.974088:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787756965.974095:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756965.974110:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10022 00000100:00080000:3.0:1787756965.974127:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756965.974139:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787756965.974200:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588654829824/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756965.974251:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756965.974256:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756965.974259:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756965.974265:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10022 00000100:00080000:3.0:1787756965.974272:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756965.974279:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756965.974291:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756965.974318:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588654829952/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756965.974341:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787756965.974354:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654829824:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787756965.974354:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756965.974361:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756965.974381:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10022 00000100:00080000:3.0:1787756965.974389:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756965.974398:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756965.974409:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756965.974432:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588654830080/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756965.974447:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756965.974452:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756965.974455:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756965.974459:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10022 00000100:00080000:3.0:1787756965.974466:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787756965.974477:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787756965.974478:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756965.974491:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756965.974507:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588654830208/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756965.974519:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756965.974524:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756965.974526:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756965.974532:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10022 00000100:00080000:3.0:1787756965.974537:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756965.974543:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756965.974550:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756965.974561:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756965/real 1787756965] req@ffff8c05a9f78380 x1874588654829824/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756965.974569:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588654830336/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756965.974628:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756965.974636:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756965.974640:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787756965.974649:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588654829824/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756965.974656:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10022 00000100:00080000:3.0:1787756965.974669:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756965.974685:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787756965.974689:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787756965.974698:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787756965.974705:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756965.974719:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654829824:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787756965.974730:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588654830464/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756965.974745:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756965.974750:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787756965.974753:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756965.974760:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10022 00000100:00080000:3.0:1787756965.974766:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787756965.974774:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654829952:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787756965.974778:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756965.974793:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756965.974811:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588654830592/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756965.974827:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756965.974832:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756965.974835:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756965.974839:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10022 00000100:00080000:3.0:1787756965.974846:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787756965.974850:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787756965.974857:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756965.974867:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756965.974955:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588654830720/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787756965.975008:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756965/real 1787756965] req@ffff8c05a9f7a680 x1874588654829952/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756965.975042:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588654829952/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756965.975062:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756965.975070:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756965.975078:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654829952:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756965.975092:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654830080:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756965.975135:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756965.975157:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756965/real 1787756965] req@ffff8c05a9f78a80 x1874588654830080/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756965.975184:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588654830080/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756965.975201:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756965.975208:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756965.975214:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654830080:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756965.975227:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654830208:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756965.975266:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756965.975288:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756965/real 1787756965] req@ffff8c05a9f7b480 x1874588654830208/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756965.975315:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588654830208/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756965.975333:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756965.975340:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756965.975347:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654830208:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756965.975359:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654830336:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756965.975395:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756965.975416:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756965/real 1787756965] req@ffff8c05a9f78700 x1874588654830336/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756965.975462:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588654830336/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756965.975477:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756965.975484:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756965.975506:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654830336:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756965.975518:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654830464:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756965.975553:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756965.975576:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756965/real 1787756965] req@ffff8c05a9f7aa00 x1874588654830464/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756965.975601:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588654830464/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756965.975634:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756965.975641:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756965.975648:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654830464:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756965.975688:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654830592:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756965.975730:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756965.975751:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756965/real 1787756965] req@ffff8c05a9f7a300 x1874588654830592/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756965.975777:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588654830592/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756965.975795:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756965.975802:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756965.975809:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654830592:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756965.975822:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654830720:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756965.975856:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756965.975891:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756965/real 1787756965] req@ffff8c05a9f79880 x1874588654830720/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756965.975997:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588654830720/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756965.976015:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756965.976022:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756965.976031:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654830720:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787756966.102013:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787756966.550092:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787756967.126057:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787756967.574129:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787756968.149982:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787756968.598197:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787756969.174015:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787756969.622220:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787756970.198039:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787756970.646137:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756971.094041:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756971.094054:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787756971.094059:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756971.094072:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10027 00000100:00080000:2.0:1787756971.094084:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756971.094093:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756971.094138:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588654830848/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756971.094175:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756971.094179:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756971.094182:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756971.094187:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10027 00000100:00080000:2.0:1787756971.094192:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756971.094196:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756971.094208:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756971.094237:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588654830976/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787756971.094255:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654830848:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787756971.094283:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756971.094292:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756971.094296:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756971.094307:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10027 00000100:00080000:2.0:1787756971.094316:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756971.094325:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756971.094344:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756971.094367:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588654831104/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787756971.094380:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787756971.094395:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756971.094401:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756971.094404:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756971.094412:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10027 00000100:00080000:2.0:1787756971.094418:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756971.094423:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756971.094431:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756971.094449:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588654831232/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756971.094460:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756971.094466:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756971.094472:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787756971.094473:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756971/real 1787756971] req@ffff8c05b2e70a80 x1874588654830848/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756971.094481:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10027 00000100:00080000:2.0:1787756971.094491:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756971.094503:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756971.094514:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756971.094528:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588654830848/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787756996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756971.094534:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588654831360/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756971.094572:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756971.094575:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787756971.094586:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787756971.094587:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756971.094595:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787756971.094598:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654830848:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787756971.094607:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10027 00000100:00080000:2.0:1787756971.094616:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756971.094621:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756971.094628:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756971.094641:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588654831488/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787756971.094646:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654830976:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787756971.094706:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787756971.094712:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787756971.094713:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787756971.094729:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756971.094738:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10027 00000100:00080000:2.0:1787756971.094745:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787756971.094755:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756971/real 1787756971] req@ffff8c05b2e72300 x1874588654830976/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756971.094756:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756971.094770:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756971.094795:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588654831616/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756971.094846:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756971.094850:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756971.094853:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756971.094859:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10027 00000100:00080000:2.0:1787756971.094864:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756971.094931:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787756971.094940:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588654830976/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787756971.094942:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756971.094968:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588654831744/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756971.094973:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756971.094996:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756971.095006:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654830976:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756971.095024:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654831104:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756971.095075:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756971.095103:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756971/real 1787756971] req@ffff8c05b2e71180 x1874588654831104/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756971.095130:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588654831104/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756971.095147:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756971.095154:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756971.095161:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654831104:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756971.095174:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654831232:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756971.095217:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756971.095242:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756971/real 1787756971] req@ffff8c05b2e72a00 x1874588654831232/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756971.095267:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588654831232/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756971.095284:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756971.095290:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756971.095297:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654831232:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756971.095310:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654831360:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756971.095347:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756971.095362:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756971/real 1787756971] req@ffff8c05b2e70e00 x1874588654831360/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756971.095376:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588654831360/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756971.095388:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756971.095392:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756971.095396:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654831360:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756971.095421:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654831488:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756971.095442:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756971.095454:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756971/real 1787756971] req@ffff8c05b2e72680 x1874588654831488/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756971.095468:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588654831488/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787756996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756971.095478:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756971.095482:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756971.095485:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654831488:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756971.095492:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654831616:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756971.095510:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756971.095521:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756971/real 1787756971] req@ffff8c05b2e72d80 x1874588654831616/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756971.095535:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588654831616/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756971.095544:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756971.095547:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756971.095552:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654831616:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756971.095558:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654831744:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756971.095576:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756971.095586:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756971/real 1787756971] req@ffff8c05b2e73100 x1874588654831744/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756971.095600:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588654831744/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787756996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756971.095610:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756971.095613:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756971.095616:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654831744:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756971.222007:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787756971.670254:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787756972.246073:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787756972.694159:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787756973.269934:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787756973.718171:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787756974.294014:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787756974.742120:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787756975.318044:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787756975.766222:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787756976.214051:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756976.214064:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787756976.214070:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756976.214083:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10032 00000100:00080000:3.0:1787756976.214096:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756976.214105:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787756976.214157:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588654831872/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756976.214198:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756976.214205:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756976.214209:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756976.214218:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10032 00000100:00080000:3.0:1787756976.214225:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756976.214233:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756976.214249:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756976.214273:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588654832000/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756976.214294:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756976.214301:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756976.214305:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787756976.214306:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654831872:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787756976.214311:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10032 00000100:00080000:3.0:1787756976.214318:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756976.214325:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756976.214334:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756976.214351:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588654832128/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756976.214369:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756976.214375:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756976.214378:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756976.214385:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10032 00000100:00080000:3.0:1787756976.214391:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756976.214397:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:0.0:1787756976.214404:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:3.0:1787756976.214406:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787756976.214442:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588654832256/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756976.214460:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787756976.214466:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756976/real 1787756976] req@ffff8c05a9f78e00 x1874588654831872/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756976.214467:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787756976.214471:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756976.214478:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10032 00000100:00080000:3.0:1787756976.214484:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756976.214491:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787756976.214497:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588654831872/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787756976.214499:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787756976.214517:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787756976.214517:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588654832384/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756976.214525:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756976.214533:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654831872:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787756976.214535:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787756976.214541:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787756976.214544:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756976.214551:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10032 00000100:00080000:3.0:1787756976.214556:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756976.214561:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787756976.214568:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654832000:192.168.201.107@tcp:38:kworker.0 00000008:00100000:3.0:1787756976.214570:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787756976.214586:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588654832512/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787756976.214608:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787756976.214609:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787756976.214614:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787756976.214618:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787756976.214625:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10032 00000100:00080000:3.0:1787756976.214631:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787756976.214632:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756976/real 1787756976] req@ffff8c05a9f79180 x1874588654832000/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787756976.214639:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756976.214648:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787756976.214656:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588654832000/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787756976.214666:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588654832640/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756976.214675:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756976.214682:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787756976.214684:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787756976.214689:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654832000:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787756976.214691:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787756976.214695:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787756976.214701:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654832128:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787756976.214701:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10032 00000100:00080000:3.0:1787756976.214707:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787756976.214750:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787756976.214759:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787756976.214773:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787756976.214779:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588654832768/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787756976.214795:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756976/real 1787756976] req@ffff8c05a9f79f80 x1874588654832128/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756976.214818:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588654832128/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756976.214835:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756976.214841:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756976.214848:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654832128:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756976.214862:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654832256:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756976.215001:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756976.215027:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756976/real 1787756976] req@ffff8c05a9f7ad80 x1874588654832256/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756976.215054:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588654832256/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756976.215073:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756976.215079:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756976.215086:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654832256:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756976.215099:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654832384:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756976.215133:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756976.215149:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756976/real 1787756976] req@ffff8c05a9f7bb80 x1874588654832384/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756976.215164:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588654832384/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756976.215174:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756976.215178:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756976.215182:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654832384:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756976.215206:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654832512:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756976.215227:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756976.215238:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756976/real 1787756976] req@ffff8c05a9f7b800 x1874588654832512/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756976.215251:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588654832512/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756976.215261:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756976.215264:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756976.215269:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654832512:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756976.215275:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654832640:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756976.215293:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756976.215304:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756976/real 1787756976] req@ffff8c05a9f7b100 x1874588654832640/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756976.215318:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588654832640/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756976.215327:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756976.215330:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756976.215334:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654832640:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756976.215340:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654832768:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756976.215357:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756976.215367:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756976/real 1787756976] req@ffff8c05a9f79500 x1874588654832768/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756976.215380:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588654832768/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756976.215389:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756976.215393:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756976.215397:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654832768:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756976.342015:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787756976.790161:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787756977.366013:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787756977.814166:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787756978.390045:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787756978.838164:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787756979.414037:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787756979.862205:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787756980.438025:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787756980.886165:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787756981.334081:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756981.334094:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787756981.334099:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756981.334115:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10037 00000100:00080000:1.0:1787756981.334132:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756981.334144:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787756981.334203:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588654832896/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756981.334269:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756981.334277:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787756981.334281:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756981.334289:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10037 00000100:00080000:1.0:1787756981.334297:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756981.334305:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756981.334320:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787756981.334350:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588654833024/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756981.334368:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756981.334373:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787756981.334376:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756981.334383:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10037 00000100:00080000:1.0:1787756981.334389:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787756981.334398:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654832896:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787756981.334399:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756981.334417:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787756981.334440:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654833152/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756981.334456:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756981.334461:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787756981.334464:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756981.334469:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10037 00000100:00080000:1.0:1787756981.334489:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787756981.334498:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787756981.334502:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756981.334513:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787756981.334532:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654833280/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756981.334545:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756981.334550:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787756981.334552:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756981.334559:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10037 00000100:00100000:0.0:1787756981.334566:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756981/real 1787756981] req@ffff8c05b2d9b480 x1874588654832896/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787756981.334587:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787756981.334594:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654832896/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787756981.334614:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787756981.334617:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787756981.334649:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787756981.334662:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756981.334670:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654832896:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787756981.334687:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654833024:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787756981.334690:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654833408/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756981.334718:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787756981.334725:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787756981.334736:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787756981.334741:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756981.334751:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10037 00000100:00100000:0.0:1787756981.334754:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756981/real 1787756981] req@ffff8c05b2d9b100 x1874588654833024/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787756981.334760:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756981.334778:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787756981.334787:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654833024/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787756981.334789:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787756981.334810:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756981.334816:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787756981.334823:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654833536/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787756981.334827:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654833024:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787756981.334860:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756981.334864:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787756981.334867:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787756981.334874:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654833152:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787756981.334927:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10037 00000100:00080000:1.0:1787756981.334934:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756981.334944:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756981.334955:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787756981.334976:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588654833664/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787756981.334984:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787756981.335007:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756981.335017:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787756981.335018:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756981/real 1787756981] req@ffff8c05b2d98380 x1874588654833152/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787756981.335021:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756981.335038:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10037 00000100:00080000:1.0:1787756981.335049:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756981.335058:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787756981.335064:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654833152/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787756981.335068:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787756981.335096:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588654833792/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787756981.335100:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756981.335113:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756981.335126:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654833152:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756981.335144:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654833280:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756981.335170:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756981.335187:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756981/real 1787756981] req@ffff8c05b2d99880 x1874588654833280/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756981.335202:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654833280/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756981.335212:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756981.335216:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756981.335220:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654833280:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756981.335227:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654833408:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756981.335245:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756981.335255:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756981/real 1787756981] req@ffff8c05b2d99f80 x1874588654833408/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756981.335269:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654833408/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756981.335278:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756981.335282:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756981.335286:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654833408:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756981.335293:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654833536:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756981.335311:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756981.335321:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756981/real 1787756981] req@ffff8c05b2d98000 x1874588654833536/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756981.335335:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654833536/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756981.335344:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756981.335347:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756981.335351:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654833536:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756981.335370:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654833664:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756981.335389:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756981.335400:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756981/real 1787756981] req@ffff8c05b2d98700 x1874588654833664/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756981.335414:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654833664/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756981.335424:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756981.335427:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756981.335431:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654833664:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756981.335437:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654833792:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756981.335454:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756981.335465:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756981/real 1787756981] req@ffff8c05b2d9a300 x1874588654833792/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756981.335479:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588654833792/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756981.335488:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756981.335492:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756981.335495:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654833792:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756981.462034:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787756981.910225:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787756982.486057:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787756982.934150:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787756983.510025:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787756983.958172:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787756984.534046:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787756984.982137:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787756985.557996:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787756986.006195:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756986.454135:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756986.454151:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787756986.454159:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756986.454176:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10042 00000100:00080000:2.0:1787756986.454192:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756986.454204:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756986.454262:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588654833920/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756986.454318:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756986.454325:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756986.454328:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756986.454336:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10042 00000100:00080000:2.0:1787756986.454346:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756986.454356:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756986.454373:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787756986.454396:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654833920:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787756986.454407:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588654834048/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756986.454434:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756986.454441:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756986.454445:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756986.454453:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10042 00000100:00080000:2.0:1787756986.454461:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756986.454469:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756986.454479:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787756986.454488:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756986.454521:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588654834176/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756986.454572:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787756986.454581:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756986/real 1787756986] req@ffff8c05b2e70000 x1874588654833920/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756986.454584:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756986.454592:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756986.454614:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10042 00000100:00100000:1.0:1787756986.454624:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588654833920/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756986.454628:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756986.454647:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:1.0:1787756986.454648:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787756986.454663:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000008:00100000:2.0:1787756986.454666:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787756986.454671:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654833920:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787756986.454702:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588654834304/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787756986.454712:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654834048:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787756986.454744:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756986.454752:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:1.0:1787756986.454759:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787756986.454762:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756986.454777:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10042 00000100:00080000:2.0:1787756986.454786:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787756986.454787:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756986/real 1787756986] req@ffff8c05b2e70700 x1874588654834048/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756986.454803:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756986.454820:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787756986.454821:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588654834048/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787756986.454842:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787756986.454847:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787756986.454857:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588654834432/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787756986.454859:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654834048:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787756986.454921:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654834176:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787756986.454943:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756986.454950:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756986.454955:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756986.454964:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10042 00000800:00000100:1.0:1787756986.454973:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787756986.454977:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756986.454992:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756986.455006:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787756986.455007:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756986/real 1787756986] req@ffff8c05b2e73480 x1874588654834176/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756986.455050:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588654834560/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787756986.455057:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588654834176/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756986.455104:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756986.455109:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756986.455114:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787756986.455123:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756986.455126:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787756986.455138:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10042 00000100:00100000:1.0:1787756986.455149:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654834176:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787756986.455152:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756986.455169:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756986.455181:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787756986.455200:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654834304:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756986.455207:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588654834688/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756986.455222:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756986.455230:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756986.455232:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756986.455242:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10042 00000100:00080000:2.0:1787756986.455249:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756986.455254:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756986.455262:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787756986.455278:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588654834816/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787756986.455307:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787756986.455342:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756986/real 1787756986] req@ffff8c05b2e71500 x1874588654834304/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787756986.455370:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588654834304/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787756986.455386:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787756986.455393:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787756986.455400:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654834304:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787756986.455413:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654834432:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787756986.455457:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787756986.455483:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756986/real 1787756986] req@ffff8c05b2e73b80 x1874588654834432/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787756986.455508:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588654834432/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787756986.455566:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787756986.455575:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787756986.455582:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654834432:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787756986.455596:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654834560:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787756986.455640:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787756986.455664:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756986/real 1787756986] req@ffff8c05b2e71c00 x1874588654834560/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787756986.455689:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588654834560/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787756986.455706:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787756986.455713:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787756986.455720:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654834560:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787756986.455745:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654834688:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787756986.455776:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787756986.455791:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756986/real 1787756986] req@ffff8c05b2e71880 x1874588654834688/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787756986.455806:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588654834688/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787756986.455817:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787756986.455821:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787756986.455826:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654834688:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787756986.455833:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654834816:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787756986.455852:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787756986.455862:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756986/real 1787756986] req@ffff8c05b2e70380 x1874588654834816/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787756986.455917:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588654834816/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787756986.455931:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787756986.455937:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787756986.455943:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654834816:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787756986.582039:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787756987.030192:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787756987.606094:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787756988.054290:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787756988.629996:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787756989.078156:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787756989.654071:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787756990.102279:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787756990.678052:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787756991.126135:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787756991.574050:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756991.574068:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787756991.574075:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756991.574103:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10047 00000100:00080000:2.0:1787756991.574121:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756991.574133:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787756991.574199:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73800 x1874588654834944/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756991.574245:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756991.574250:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756991.574253:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756991.574258:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10047 00000100:00080000:2.0:1787756991.574263:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756991.574268:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756991.574282:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756991.574300:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588654835072/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756991.574313:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756991.574318:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756991.574321:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756991.574327:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10047 00000100:00080000:2.0:1787756991.574330:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756991.574334:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756991.574339:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787756991.574351:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588654835200/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756991.574364:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756991.574371:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756991.574374:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756991.574380:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10047 00000100:00080000:2.0:1787756991.574385:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756991.574390:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756991.574396:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787756991.574408:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654834944:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787756991.574465:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588654835328/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756991.574506:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756991.574514:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787756991.574518:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756991.574526:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10047 00000100:00080000:2.0:1787756991.574535:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756991.574545:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756991.574555:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787756991.574576:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756991.574577:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588654835456/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756991.574598:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756991.574603:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787756991.574607:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756991.574612:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10047 00000100:00080000:2.0:1787756991.574618:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756991.574626:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756991.574634:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787756991.574647:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756991/real 1787756991] req@ffff8c05b2e73800 x1874588654834944/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787756991.574654:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588654835584/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756991.574716:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756991.574748:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787756991.574757:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73800 x1874588654834944/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787756991.574758:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756991.574785:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10047 00000100:00080000:2.0:1787756991.574803:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756991.574807:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787756991.574827:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:1.0:1787756991.574843:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000008:00100000:2.0:1787756991.574849:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787756991.574857:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654834944:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787756991.574884:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654835072:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787756991.574936:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588654835712/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787756991.574955:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787756991.574959:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787756991.574962:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787756991.574967:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10047 00000100:00080000:2.0:1787756991.574976:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787756991.574983:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787756991.574991:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787756991.575024:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787756991.575025:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588654835840/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787756991.575067:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756991/real 1787756991] req@ffff8c05b2e70380 x1874588654835072/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787756991.575105:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588654835072/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787756991.575116:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787756991.575120:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787756991.575124:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654835072:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787756991.575133:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654835200:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787756991.575161:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787756991.575172:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756991/real 1787756991] req@ffff8c05b2e71880 x1874588654835200/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787756991.575185:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588654835200/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787756991.575193:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787756991.575197:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787756991.575200:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654835200:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787756991.575231:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654835328:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787756991.575250:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787756991.575260:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756991/real 1787756991] req@ffff8c05b2e71c00 x1874588654835328/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787756991.575273:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588654835328/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787756991.575282:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787756991.575285:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787756991.575288:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654835328:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787756991.575295:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654835456:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787756991.575312:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787756991.575321:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756991/real 1787756991] req@ffff8c05b2e73b80 x1874588654835456/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787756991.575333:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588654835456/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787756991.575341:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787756991.575344:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787756991.575347:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654835456:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787756991.575353:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654835584:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787756991.575369:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787756991.575378:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756991/real 1787756991] req@ffff8c05b2e71500 x1874588654835584/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787756991.575389:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588654835584/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787756991.575416:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787756991.575420:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787756991.575423:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654835584:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787756991.575478:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654835712:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787756991.575523:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787756991.575536:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756991/real 1787756991] req@ffff8c05b2e73480 x1874588654835712/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787756991.575549:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588654835712/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787756991.575558:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787756991.575562:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787756991.575565:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654835712:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787756991.575571:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654835840:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787756991.575588:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787756991.575598:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756991/real 1787756991] req@ffff8c05b2e70700 x1874588654835840/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787756991.575610:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588654835840/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787756991.575618:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787756991.575621:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787756991.575625:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654835840:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756991.702130:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787756992.150323:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787756992.726090:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787756993.174125:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787756993.750079:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787756994.198234:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787756994.774070:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787756995.222160:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787756995.798027:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787756996.246246:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787756996.694110:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756996.694126:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787756996.694133:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756996.694150:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10053 00000100:00080000:1.0:1787756996.694169:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756996.694181:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787756996.694249:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588654835968/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756996.694299:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756996.694304:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787756996.694307:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756996.694312:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10053 00000100:00080000:1.0:1787756996.694317:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756996.694322:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756996.694336:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787756996.694393:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588654836096/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756996.694410:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756996.694419:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787756996.694423:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654835968:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787756996.694424:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756996.694440:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10053 00000100:00080000:1.0:1787756996.694447:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756996.694451:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756996.694457:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787756996.694479:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588654836224/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756996.694494:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756996.694499:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787756996.694502:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756996.694508:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10053 00000100:00080000:1.0:1787756996.694529:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756996.694534:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756996.694541:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787756996.694549:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787756996.694566:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588654836352/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756996.694582:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756996.694587:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787756996.694589:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756996.694593:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10053 00000100:00080000:1.0:1787756996.694598:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756996.694604:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756996.694611:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787756996.694628:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756996/real 1787756996] req@ffff8c05b2d9aa00 x1874588654835968/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787756996.694633:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654836480/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787756996.694690:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588654835968/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787756996.694691:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787756996.694707:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787756996.694714:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756996.694730:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10053 00000100:00080000:0.0:1787756996.694732:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787756996.694738:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787756996.694746:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787756996.694754:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787756996.694761:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654835968:192.168.201.107@tcp:250:kworker.0 00000008:00100000:1.0:1787756996.694765:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787756996.694787:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654836608/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787756996.694805:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787756996.694811:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654836096:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787756996.694813:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787756996.694821:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756996.694830:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10053 00000100:00080000:1.0:1787756996.694835:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756996.694842:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787756996.694849:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787756996.694857:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787756996.694871:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654836736/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787756996.694928:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756996/real 1787756996] req@ffff8c05b2d9bb80 x1874588654836096/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787756996.694987:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787756996.694996:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588654836096/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787756996.694998:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787756996.695004:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787756996.695018:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10053 00000100:00080000:0.0:1787756996.695030:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787756996.695030:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787756996.695042:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787756996.695052:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000008:00100000:1.0:1787756996.695054:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787756996.695066:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654836096:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787756996.695079:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654836864/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787756996.695091:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654836224:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756996.695149:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756996.695172:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756996/real 1787756996] req@ffff8c05b2d9a300 x1874588654836224/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756996.695197:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588654836224/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756996.695214:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756996.695221:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756996.695228:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654836224:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756996.695242:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654836352:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756996.695283:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756996.695299:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756996/real 1787756996] req@ffff8c05b2d98700 x1874588654836352/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756996.695316:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654836352/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756996.695327:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756996.695330:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756996.695335:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654836352:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756996.695342:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654836480:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756996.695401:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756996.695416:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756996/real 1787756996] req@ffff8c05b2d98000 x1874588654836480/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756996.695431:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654836480/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756996.695442:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756996.695446:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756996.695450:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654836480:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756996.695477:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654836608:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787756996.695500:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756996.695512:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756996/real 1787756996] req@ffff8c05b2d99f80 x1874588654836608/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756996.695526:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654836608/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756996.695545:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756996.695549:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756996.695553:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654836608:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787756996.695560:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654836736:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756996.695579:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756996.695590:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756996/real 1787756996] req@ffff8c05b2d99880 x1874588654836736/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756996.695604:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654836736/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756996.695614:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756996.695618:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756996.695621:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654836736:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756996.695628:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654836864:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787756996.695645:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787756996.695657:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787756996/real 1787756996] req@ffff8c05b2d98380 x1874588654836864/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787756996.695671:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654836864/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787756996.695680:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787756996.695683:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787756996.695688:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654836864:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787756996.822007:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787756997.270157:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787756997.846120:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787756998.294221:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787756998.869984:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787756999.318113:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787756999.894037:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757000.342237:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757000.918003:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757001.366118:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757001.814106:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757001.814134:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787757001.814141:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757001.814158:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10058 00000100:00080000:0.0:1787757001.814185:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757001.814198:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757001.814248:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588654836992/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757001.814324:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757001.814332:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757001.814336:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757001.814346:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10058 00000100:00080000:0.0:1787757001.814355:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757001.814365:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757001.814381:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757001.814411:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654836992:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787757001.814412:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588654837120/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757001.814441:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757001.814449:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757001.814452:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757001.814458:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10058 00000100:00080000:0.0:1787757001.814465:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757001.814469:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757001.814477:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757001.814493:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588654837248/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757001.814509:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757001.814514:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757001.814516:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757001.814521:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10058 00000100:00080000:0.0:1787757001.814526:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757001.814531:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757001.814537:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757001.814555:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588654837376/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757001.814568:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757001.814573:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757001.814576:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757001.814581:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10058 00000100:00100000:1.0:1787757001.814588:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654837120:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757001.814590:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757001.814605:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757001.814616:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757001.814636:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588654837504/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757001.814651:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654837248:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757001.814653:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757001.814667:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757001.814669:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757001.814679:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10058 00000100:00100000:1.0:1787757001.814702:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654837376:192.168.201.107@tcp:8:kworker.0 00000100:00080000:0.0:1787757001.814705:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757001.814717:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757001.814727:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757001.814748:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654837504:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757001.814754:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588654837632/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757001.814774:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757001.814778:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787757001.814781:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757001.814788:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10058 00000100:00080000:0.0:1787757001.814794:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787757001.814796:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654837632:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757001.814807:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757001.814818:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757001.814860:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588654837760/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757001.814946:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757001.814954:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757001.814958:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757001.814967:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10058 00000100:00080000:0.0:1787757001.814976:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757001.814986:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757001.814998:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757001.815008:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654837760:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757001.815029:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588654837888/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757001.815085:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654837888:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757001.942035:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757002.390218:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757002.966077:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757003.414101:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757003.990011:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757004.438130:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787757004.886419:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.107:988 00000400:00000100:2.0:1787757004.886499:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.107@tcp at host 192.168.201.107 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787757004.886528:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787757004.886610:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787757004.886629:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787757004.886642:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787757004.886657:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787757004.886674:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787757004.886691:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787757004.886712:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:1.0:1787757004.886744:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757001/real 1787757004] req@ffff8c05a9f77800 x1874588654837760/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757004.886794:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588654837760/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757004.886832:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757004.886842:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757004.886850:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654837760:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757004.886866:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757001/real 1787757004] req@ffff8c05a9f76300 x1874588654837632/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757004.886942:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588654837632/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757004.886958:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757004.886964:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757004.886970:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654837632:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757004.886982:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757001/real 1787757004] req@ffff8c05a9f74e00 x1874588654837504/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757004.887005:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588654837504/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757004.887020:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757004.887026:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757004.887032:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654837504:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757004.887044:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757001/real 1787757004] req@ffff8c05a9f74000 x1874588654837120/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757004.887070:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588654837120/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757004.887085:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757004.887092:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757004.887098:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654837120:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757004.887110:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757001/real 1787757004] req@ffff8c05a9f75880 x1874588654837248/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757004.887132:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588654837248/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757004.887147:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757004.887153:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757004.887160:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654837248:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757004.887173:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757001/real 1787757004] req@ffff8c05a9f75500 x1874588654837376/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757004.887195:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588654837376/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757004.887241:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757004.887249:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757004.887257:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654837376:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757004.887275:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757001/real 1787757004] req@ffff8c05a9f77100 x1874588654836992/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757004.887302:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588654836992/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757004.887320:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757004.887326:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757004.887332:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654836992:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787757004.887413:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757001/real 1787757004] req@ffff8c05a9f75180 x1874588654837888/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757004.887436:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588654837888/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757004.887450:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757004.887457:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757004.887463:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654837888:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757005.014038:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757005.462250:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757006.038034:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757006.486247:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757006.934099:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757006.934111:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787757006.934116:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757006.934127:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10063 00000100:00080000:1.0:1787757006.934140:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757006.934148:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757006.934229:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588654838016/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757006.934267:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757006.934272:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757006.934275:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757006.934280:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10063 00000100:00080000:1.0:1787757006.934284:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757006.934289:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757006.934318:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757006.934337:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588654838144/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757006.934353:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757006.934358:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757006.934361:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757006.934387:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654838016:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757006.934388:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10063 00000100:00080000:1.0:1787757006.934395:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757006.934405:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757006.934413:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757006.934428:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654838272/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757006.934440:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757006.934445:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757006.934447:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757006.934451:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10063 00000100:00080000:1.0:1787757006.934501:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757006.934515:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:0.0:1787757006.934517:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:1.0:1787757006.934532:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757006.934560:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654838400/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757006.934590:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757006/real 1787757006] req@ffff8c05b2d9b100 x1874588654838016/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757006.934607:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757006.934617:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757006.934622:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757006.934634:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10063 00000100:00100000:0.0:1787757006.934635:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654838016/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757006.934645:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757006.934661:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787757006.934670:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787757006.934673:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787757006.934686:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757006.934697:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654838016:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787757006.934706:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654838528/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757006.934726:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757006.934732:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757006.934736:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757006.934747:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10063 00000100:00100000:0.0:1787757006.934748:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654838144:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787757006.934755:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757006.934767:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757006.934778:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757006.934804:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654838656/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757006.934818:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757006.934837:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757006.934844:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787757006.934848:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757006.934857:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757006/real 1787757006] req@ffff8c05b2d9b480 x1874588654838144/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757006.934864:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10063 00000100:00100000:0.0:1787757006.934939:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654838144/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757006.934964:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757006.934972:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757006.934979:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787757006.934988:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000008:00100000:1.0:1787757006.934996:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757006.935002:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654838144:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757006.935021:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654838272:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757006.935028:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588654838784/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757006.935051:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757006.935058:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757006.935062:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787757006.935068:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757006.935077:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10063 00000100:00080000:1.0:1787757006.935088:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757006.935100:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757006/real 1787757006] req@ffff8c05b2d98380 x1874588654838272/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757006.935101:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757006.935117:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757006.935145:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654838272/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757006.935164:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787757006.935168:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588654838912/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757006.935205:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757006.935218:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654838272:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757006.935232:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654838400:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757006.935277:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757006.935309:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757006/real 1787757006] req@ffff8c05b2d99880 x1874588654838400/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757006.935332:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654838400/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757006.935347:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757006.935353:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757006.935360:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654838400:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757006.935372:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654838528:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757006.935408:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757006.935429:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757006/real 1787757006] req@ffff8c05b2d99f80 x1874588654838528/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757006.935448:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654838528/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757006.935463:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757006.935467:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757006.935473:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654838528:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757006.935508:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654838656:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757006.935546:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757006.935569:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757006/real 1787757006] req@ffff8c05b2d98000 x1874588654838656/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757006.935595:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654838656/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757006.935621:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757006.935628:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757006.935634:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654838656:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757006.935647:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654838784:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757006.935685:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757006.935707:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757006/real 1787757006] req@ffff8c05b2d98700 x1874588654838784/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757006.935730:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654838784/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757006.935745:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757006.935751:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757006.935757:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654838784:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757006.935767:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654838912:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757006.935799:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757006.935818:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757006/real 1787757006] req@ffff8c05b2d9a300 x1874588654838912/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757006.935839:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588654838912/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757006.935853:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757006.935859:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757006.935864:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654838912:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757007.062071:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757007.510210:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757008.086030:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757008.534315:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757009.110006:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757009.558119:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757010.133987:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757010.582277:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757011.157982:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757011.606149:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757012.054009:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757012.054022:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787757012.054028:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757012.054040:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10068 00000100:00080000:2.0:1787757012.054055:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757012.054068:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757012.054162:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588654839040/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757012.054209:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757012.054217:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757012.054221:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757012.054228:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10068 00000100:00080000:2.0:1787757012.054237:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757012.054247:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757012.054264:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757012.054297:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654839040:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787757012.054300:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588654839168/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757012.054340:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757012.054346:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757012.054348:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757012.054355:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10068 00000100:00080000:2.0:1787757012.054362:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757012.054367:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757012.054390:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787757012.054404:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757012.054414:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588654839296/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757012.054432:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757012.054436:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757012.054439:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757012.054444:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10068 00000100:00080000:2.0:1787757012.054448:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757012.054453:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757012.054460:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757012.054483:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757012/real 1787757012] req@ffff8c05b2e70000 x1874588654839040/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757012.054494:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588654839424/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757012.054541:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787757012.054546:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588654839040/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757012.054555:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757012.054564:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757012.054578:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757012.054582:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10068 00000100:00080000:0.0:1787757012.054590:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787757012.054597:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757012.054599:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654839040:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757012.054611:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757012.054622:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757012.054641:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654839168:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757012.054649:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588654839552/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757012.054668:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757012.054673:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757012.054676:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757012.054682:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10068 00000100:00080000:2.0:1787757012.054688:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787757012.054692:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757012.054703:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757012.054714:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757012.054739:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757012/real 1787757012] req@ffff8c05b2e73100 x1874588654839168/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757012.054746:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588654839680/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757012.054777:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588654839168/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757012.054795:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757012.054805:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787757012.054805:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757012.054816:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787757012.054822:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654839168:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757012.054823:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757012.054840:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10068 00000100:00080000:2.0:1787757012.054847:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757012.054855:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654839296:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757012.054859:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757012.054913:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757012.054936:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588654839808/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757012.054960:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757012.054966:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757012.054969:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787757012.054978:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757012.054981:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10068 00000100:00080000:2.0:1787757012.054992:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757012.054998:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757012.055006:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757012.055008:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757012/real 1787757012] req@ffff8c05b2e72d80 x1874588654839296/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757012.055029:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588654839936/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757012.055036:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588654839296/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757012.055078:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757012.055115:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757012.055121:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654839296:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757012.055136:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654839424:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757012.055167:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757012.055182:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757012/real 1787757012] req@ffff8c05b2e72680 x1874588654839424/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757012.055197:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588654839424/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757012.055208:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757012.055212:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757012.055216:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654839424:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757012.055238:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654839552:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757012.055259:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757012.055271:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757012/real 1787757012] req@ffff8c05b2e70e00 x1874588654839552/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757012.055285:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588654839552/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757012.055295:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757012.055299:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757012.055302:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654839552:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757012.055309:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654839680:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757012.055327:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757012.055338:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757012/real 1787757012] req@ffff8c05b2e72a00 x1874588654839680/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757012.055353:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588654839680/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757012.055363:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757012.055366:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757012.055369:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654839680:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757012.055376:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654839808:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757012.055406:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757012.055426:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757012/real 1787757012] req@ffff8c05b2e71180 x1874588654839808/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757012.055450:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588654839808/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757012.055467:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757012.055473:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757012.055481:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654839808:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757012.055493:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654839936:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757012.055524:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757012.055538:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757012/real 1787757012] req@ffff8c05b2e72300 x1874588654839936/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757012.055553:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588654839936/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757012.055571:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757012.055575:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757012.055579:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654839936:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757012.182028:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757012.630289:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757013.205982:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757013.654248:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757014.230060:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757014.678377:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757015.254159:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757015.702198:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757016.278038:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757016.726310:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757017.174194:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757017.174210:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787757017.174218:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757017.174235:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10073 00000100:00080000:1.0:1787757017.174251:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757017.174274:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757017.174338:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588654840064/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757017.174373:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757017.174377:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757017.174380:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757017.174385:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10073 00000100:00080000:1.0:1787757017.174396:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757017.174403:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757017.174414:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757017.174431:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588654840192/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757017.174445:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787757017.174454:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654840064:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757017.174455:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757017.174461:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757017.174475:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10073 00000100:00080000:1.0:1787757017.174478:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757017.174482:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757017.174488:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757017.174503:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588654840320/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757017.174523:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757017.174527:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757017.174530:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757017.174535:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10073 00000100:00080000:1.0:1787757017.174539:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757017.174543:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757017.174558:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787757017.174563:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757017.174578:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588654840448/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757017.174592:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757017.174596:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757017.174598:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757017.174604:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10073 00000100:00080000:1.0:1787757017.174611:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757017.174619:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757017.174625:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757017/real 1787757017] req@ffff8c05b2d9bb80 x1874588654840064/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787757017.174631:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757017.174673:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588654840064/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757017.174678:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654840576/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757017.174729:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757017.174731:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757017.174761:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757017.174767:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757017.174769:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787757017.174778:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10073 00000100:00080000:1.0:1787757017.174789:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757017.174791:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654840064:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757017.174802:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757017.174813:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757017.174832:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654840704/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757017.174841:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654840192:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787757017.174865:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757017.174871:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787757017.174874:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757017.174881:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10073 00000100:00080000:1.0:1787757017.174967:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787757017.174976:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757017.174982:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757017.175042:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757017.175063:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757017/real 1787757017] req@ffff8c05b2d9aa00 x1874588654840192/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757017.175069:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654840832/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757017.175116:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787757017.175122:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588654840192/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757017.175124:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757017.175130:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757017.175144:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10073 00000100:00080000:0.0:1787757017.175154:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757017.175155:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757017.175171:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787757017.175171:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757017.175185:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757017.175187:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654840192:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757017.175205:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654840320:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757017.175210:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654840960/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757017.175281:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757017.175301:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757017/real 1787757017] req@ffff8c05b2d9a300 x1874588654840320/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757017.175318:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588654840320/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757017.175330:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757017.175334:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757017.175339:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654840320:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757017.175346:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654840448:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757017.175368:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757017.175380:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757017/real 1787757017] req@ffff8c05b2d98700 x1874588654840448/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757017.175394:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654840448/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757017.175404:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757017.175408:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757017.175412:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654840448:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757017.175419:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654840576:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757017.175437:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757017.175466:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757017/real 1787757017] req@ffff8c05b2d98000 x1874588654840576/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757017.175480:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654840576/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757017.175489:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757017.175492:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757017.175496:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654840576:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757017.175517:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654840704:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757017.175537:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757017.175548:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757017/real 1787757017] req@ffff8c05b2d99f80 x1874588654840704/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757017.175561:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654840704/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757017.175570:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757017.175573:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757017.175577:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654840704:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757017.175583:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654840832:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757017.175600:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757017.175610:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757017/real 1787757017] req@ffff8c05b2d99880 x1874588654840832/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757017.175623:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654840832/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757017.175640:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757017.175643:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757017.175647:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654840832:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757017.175653:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654840960:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757017.175676:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757017.175694:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757017/real 1787757017] req@ffff8c05b2d98380 x1874588654840960/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757017.175717:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654840960/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757017.175733:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757017.175739:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757017.175746:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654840960:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757017.302039:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757017.750202:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757018.326023:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757018.774260:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757019.350040:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757019.798146:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757020.374035:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757020.822227:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757021.398068:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757021.846159:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757022.294156:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757022.294167:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787757022.294173:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757022.294189:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10078 00000100:00080000:2.0:1787757022.294202:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757022.294212:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757022.294261:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588654841088/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757022.294304:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757022.294309:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757022.294312:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757022.294317:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10078 00000100:00080000:2.0:1787757022.294322:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757022.294327:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757022.294341:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757022.294362:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588654841216/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757022.294378:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757022.294383:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757022.294387:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757022.294394:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10078 00000100:00080000:2.0:1787757022.294400:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757022.294407:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757022.294416:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757022.294432:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588654841344/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757022.294445:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654841088:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757022.294453:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757022.294460:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757022.294463:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757022.294470:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10078 00000100:00080000:2.0:1787757022.294477:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757022.294481:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757022.294488:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757022.294504:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588654841472/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757022.294515:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757022.294520:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757022.294524:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757022.294528:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10078 00000100:00080000:2.0:1787757022.294533:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787757022.294542:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757022.294543:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757022.294555:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757022.294571:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588654841600/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757022.294584:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757022.294588:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757022.294591:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757022.294596:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10078 00000100:00080000:2.0:1787757022.294601:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787757022.294606:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757022/real 1787757022] req@ffff8c05b2e70a80 x1874588654841088/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757022.294612:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757022.294625:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757022.294651:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588654841728/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757022.294656:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588654841088/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757022.294690:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757022.294697:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757022.294698:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787757022.294706:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757022.294712:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787757022.294719:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10078 00000100:00100000:1.0:1787757022.294724:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654841088:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757022.294729:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757022.294738:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757022.294742:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654841216:192.168.201.107@tcp:38:kworker.0 00000008:00100000:2.0:1787757022.294749:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757022.294773:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588654841856/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787757022.294788:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757022.294792:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757022.294799:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757022.294803:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757022.294811:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10078 00000100:00100000:1.0:1787757022.294817:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757022/real 1787757022] req@ffff8c05b2e72300 x1874588654841216/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757022.294822:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757022.294836:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757022.294849:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757022.294858:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588654841216/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757022.294943:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787757022.294952:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588654841984/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757022.294957:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757022.294976:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654841216:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757022.295043:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654841344:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757022.295104:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757022.295133:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757022/real 1787757022] req@ffff8c05b2e71180 x1874588654841344/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757022.295157:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588654841344/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757022.295168:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757022.295174:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757022.295180:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654841344:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757022.295189:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654841472:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757022.295218:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757022.295245:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757022/real 1787757022] req@ffff8c05b2e72a00 x1874588654841472/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757022.295269:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588654841472/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757022.295279:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757022.295284:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757022.295290:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654841472:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757022.295299:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654841600:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757022.295324:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757022.295348:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757022/real 1787757022] req@ffff8c05b2e70e00 x1874588654841600/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757022.295373:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588654841600/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757022.295383:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757022.295386:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757022.295390:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654841600:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757022.295397:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654841728:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757022.295415:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757022.295426:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757022/real 1787757022] req@ffff8c05b2e72680 x1874588654841728/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757022.295440:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588654841728/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757022.295450:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757022.295453:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757022.295457:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654841728:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757022.295463:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654841856:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757022.295481:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757022.295492:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757022/real 1787757022] req@ffff8c05b2e72d80 x1874588654841856/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757022.295505:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588654841856/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757022.295523:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757022.295527:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757022.295531:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654841856:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757022.295537:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654841984:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757022.295554:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757022.295564:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757022/real 1787757022] req@ffff8c05b2e73100 x1874588654841984/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757022.295577:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588654841984/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757022.295587:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757022.295591:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757022.295594:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654841984:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757022.422004:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757022.870194:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757023.445996:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757023.894037:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757024.470001:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757024.918211:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757025.494008:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757025.942070:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757026.517975:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757026.966217:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757027.414015:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757027.414027:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787757027.414033:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757027.414047:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10083 00000100:00080000:2.0:1787757027.414060:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757027.414069:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757027.414127:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588654842112/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757027.414176:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757027.414184:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757027.414187:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757027.414196:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10083 00000100:00080000:2.0:1787757027.414205:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757027.414212:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757027.414225:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757027.414264:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654842112:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787757027.414271:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588654842240/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757027.414298:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757027.414304:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757027.414308:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757027.414317:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10083 00000100:00080000:2.0:1787757027.414326:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757027.414335:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757027.414345:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757027.414367:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588654842368/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787757027.414372:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757027.414390:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757027.414396:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757027.414399:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757027.414406:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10083 00000100:00080000:2.0:1787757027.414412:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757027.414418:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757027.414426:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757027.414441:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588654842496/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757027.414451:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757027/real 1787757027] req@ffff8c05b2e70000 x1874588654842112/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757027.414464:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757027.414479:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757027.414485:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757027.414498:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10083 00000100:00080000:2.0:1787757027.414504:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757027.414509:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757027.414518:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588654842112/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787757027.414520:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757027.414540:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588654842624/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757027.414579:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757027.414584:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757027.414586:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757027.414590:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10083 00000100:00080000:2.0:1787757027.414600:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757027.414613:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:1.0:1787757027.414619:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787757027.414620:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787757027.414635:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757027.414643:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588654842752/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757027.414648:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654842112:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757027.414673:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757027.414678:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787757027.414681:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757027.414686:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10083 00000100:00080000:2.0:1787757027.414692:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787757027.414698:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654842240:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757027.414700:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757027.414715:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757027.414734:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588654842880/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757027.414747:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757027.414752:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757027.414754:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787757027.414766:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757027.414766:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10083 00000100:00080000:2.0:1787757027.414779:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757027.414785:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757027.414791:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757027.414838:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588654843008/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757027.414861:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757027/real 1787757027] req@ffff8c05b2e70700 x1874588654842240/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757027.414953:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588654842240/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757027.414965:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757027.414970:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757027.414975:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654842240:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757027.414984:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654842368:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757027.415018:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757027.415035:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757027/real 1787757027] req@ffff8c05b2e73480 x1874588654842368/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757027.415050:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588654842368/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757027.415060:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757027.415064:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757027.415068:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654842368:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757027.415077:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654842496:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757027.415098:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757027.415110:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757027/real 1787757027] req@ffff8c05b2e71500 x1874588654842496/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757027.415124:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588654842496/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757027.415134:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757027.415138:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757027.415141:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654842496:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757027.415148:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654842624:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757027.415167:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757027.415178:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757027/real 1787757027] req@ffff8c05b2e73b80 x1874588654842624/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757027.415192:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588654842624/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757027.415202:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757027.415205:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757027.415208:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654842624:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757027.415215:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654842752:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757027.415232:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757027.415243:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757027/real 1787757027] req@ffff8c05b2e71c00 x1874588654842752/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757027.415258:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588654842752/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757027.415268:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757027.415271:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757027.415275:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654842752:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757027.415299:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654842880:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757027.415318:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757027.415330:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757027/real 1787757027] req@ffff8c05b2e71880 x1874588654842880/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757027.415343:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588654842880/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757027.415353:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757027.415356:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757027.415360:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654842880:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757027.415367:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654843008:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757027.415385:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757027.415396:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757027/real 1787757027] req@ffff8c05b2e70380 x1874588654843008/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757027.415409:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588654843008/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757027.415428:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757027.415432:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757027.415435:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654843008:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757027.542000:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757027.990077:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757028.565997:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757029.014221:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757029.590033:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757030.038102:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757030.613997:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757031.062174:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757031.638005:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757032.086128:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757032.534077:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757032.534094:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787757032.534101:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757032.534116:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10088 00000100:00080000:0.0:1787757032.534133:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757032.534145:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757032.534211:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588654843136/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757032.534256:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757032.534262:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757032.534265:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757032.534271:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10088 00000100:00080000:0.0:1787757032.534276:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757032.534280:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757032.534296:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757032.534326:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588654843264/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757032.534349:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757032.534356:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757032.534360:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757032.534369:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10088 00000100:00080000:0.0:1787757032.534377:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787757032.534392:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654843136:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757032.534394:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757032.534413:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757032.534440:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588654843392/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757032.534466:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757032.534474:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757032.534478:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757032.534487:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10088 00000100:00080000:0.0:1787757032.534514:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787757032.534520:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757032.534533:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757032.534546:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757032.534570:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588654843520/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757032.534586:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787757032.534592:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757032/real 1787757032] req@ffff8c05a9f75180 x1874588654843136/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757032.534594:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757032.534601:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757032.534625:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10088 00000100:00080000:0.0:1787757032.534643:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757032.534652:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757032.534660:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588654843136/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787757032.534661:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757032.534697:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588654843648/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757032.534713:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757032.534716:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757032.534754:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757032.534761:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787757032.534765:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787757032.534774:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654843136:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757032.534786:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10088 00000100:00100000:1.0:1787757032.534795:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654843264:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757032.534801:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757032.534817:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:1.0:1787757032.534842:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:0.0:1787757032.534843:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757032.534870:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757032/real 1787757032] req@ffff8c05a9f77100 x1874588654843264/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757032.534872:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588654843776/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757032.534986:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588654843264/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757032.535024:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757032.535030:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787757032.535034:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757032.535050:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10088 00000100:00080000:0.0:1787757032.535064:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757032.535080:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757032.535101:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787757032.535125:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787757032.535127:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588654843904/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757032.535154:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787757032.535158:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757032.535171:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757032.535175:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787757032.535179:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654843264:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757032.535185:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10088 00000100:00080000:0.0:1787757032.535196:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757032.535211:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757032.535218:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757032.535233:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588654844032/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757032.535262:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654843392:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757032.535318:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757032.535346:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757032/real 1787757032] req@ffff8c05a9f75500 x1874588654843392/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757032.535364:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588654843392/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757032.535382:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757032.535389:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757032.535396:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654843392:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757032.535409:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654843520:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757032.535452:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757032.535476:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757032/real 1787757032] req@ffff8c05a9f75880 x1874588654843520/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757032.535495:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588654843520/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757032.535506:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757032.535510:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757032.535515:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654843520:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757032.535524:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654843648:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757032.535552:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757032.535565:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757032/real 1787757032] req@ffff8c05a9f74000 x1874588654843648/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757032.535580:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588654843648/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757032.535590:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757032.535594:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757032.535597:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654843648:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757032.535604:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654843776:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757032.535623:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757032.535635:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757032/real 1787757032] req@ffff8c05a9f74e00 x1874588654843776/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757032.535649:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588654843776/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757032.535658:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757032.535662:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757032.535666:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654843776:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757032.535673:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654843904:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757032.535691:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757032.535703:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757032/real 1787757032] req@ffff8c05a9f76300 x1874588654843904/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757032.535717:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588654843904/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757032.535764:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757032.535769:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757032.535774:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654843904:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757032.535782:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654844032:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757032.535811:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757032.535827:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757032/real 1787757032] req@ffff8c05a9f77800 x1874588654844032/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757032.535841:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588654844032/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757032.535851:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757032.535855:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757032.535859:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654844032:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757032.662025:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757033.110267:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757033.686027:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757034.134120:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757034.710058:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757035.158240:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757035.734038:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757036.182174:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757036.758011:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757037.206279:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757037.654152:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757037.654166:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787757037.654171:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757037.654184:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10094 00000100:00080000:2.0:1787757037.654197:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757037.654209:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757037.654281:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73800 x1874588654844160/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757037.654321:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757037.654324:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757037.654327:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757037.654331:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10094 00000100:00080000:2.0:1787757037.654344:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757037.654350:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757037.654361:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757037.654380:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588654844288/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757037.654391:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654844160:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757037.654397:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757037.654404:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757037.654407:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757037.654415:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10094 00000100:00080000:2.0:1787757037.654421:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757037.654427:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757037.654437:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757037.654458:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588654844416/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787757037.654471:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757037.654495:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757037.654501:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757037.654505:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757037.654513:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10094 00000100:00080000:2.0:1787757037.654520:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757037.654527:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757037.654536:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757037/real 1787757037] req@ffff8c05b2e73800 x1874588654844160/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787757037.654538:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757037.654569:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588654844544/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757037.654575:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73800 x1874588654844160/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757037.654617:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757037.654621:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757037.654624:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787757037.654652:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1787757037.654654:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654844160:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757037.654659:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757037.654672:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10094 00000100:00080000:2.0:1787757037.654683:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757037.654689:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757037.654698:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757037.654701:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654844288:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757037.654721:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588654844672/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787757037.654738:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757037.654742:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757037.654750:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757037.654753:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757037.654762:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10094 00000100:00100000:1.0:1787757037.654768:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757037/real 1787757037] req@ffff8c05b2e70380 x1874588654844288/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757037.654773:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757037.654788:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757037.654793:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588654844288/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787757037.654800:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787757037.654812:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757037.654821:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757037.654827:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588654844800/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757037.654834:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654844288:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757037.654851:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757037.654855:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787757037.654860:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654844416:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757037.654861:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757037.654873:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10094 00000100:00080000:2.0:1787757037.654936:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757037.654944:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757037.654952:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757037.654987:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588654844928/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757037.655001:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757037.655005:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757037.655010:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787757037.655011:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757037.655021:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10094 00000100:00080000:2.0:1787757037.655028:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787757037.655055:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757037/real 1787757037] req@ffff8c05b2e71880 x1874588654844416/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757037.655055:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757037.655071:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757037.655096:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588654844416/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757037.655102:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588654845056/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757037.655180:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757037.655193:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757037.655201:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654844416:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757037.655217:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654844544:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757037.655274:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757037.655291:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757037/real 1787757037] req@ffff8c05b2e71c00 x1874588654844544/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757037.655310:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588654844544/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757037.655323:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757037.655326:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757037.655330:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654844544:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757037.655348:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654844672:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757037.655370:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757037.655381:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757037/real 1787757037] req@ffff8c05b2e73b80 x1874588654844672/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757037.655393:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588654844672/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757037.655401:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757037.655404:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757037.655408:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654844672:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757037.655413:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654844800:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757037.655427:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757037.655436:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757037/real 1787757037] req@ffff8c05b2e71500 x1874588654844800/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757037.655446:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588654844800/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757037.655453:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757037.655456:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757037.655459:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654844800:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757037.655464:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654844928:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757037.655477:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757037.655485:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757037/real 1787757037] req@ffff8c05b2e73480 x1874588654844928/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757037.655496:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588654844928/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757037.655503:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757037.655505:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757037.655508:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654844928:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757037.655513:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654845056:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757037.655526:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757037.655535:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757037/real 1787757037] req@ffff8c05b2e70700 x1874588654845056/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757037.655545:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588654845056/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757037.655553:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757037.655555:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757037.655558:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654845056:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757037.782002:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757038.230158:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757038.806000:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757039.254273:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757039.830004:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757040.278073:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757040.854037:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757041.302312:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757041.878007:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757042.326104:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757042.774077:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757042.774089:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787757042.774095:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757042.774107:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10099 00000100:00080000:1.0:1787757042.774120:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757042.774128:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757042.774171:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588654845184/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757042.774209:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757042.774214:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757042.774216:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757042.774221:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10099 00000100:00080000:1.0:1787757042.774225:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757042.774231:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757042.774242:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757042.774279:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588654845312/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757042.774291:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654845184:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757042.774316:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757042.774324:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757042.774329:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757042.774339:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10099 00000100:00080000:1.0:1787757042.774349:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757042.774356:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757042.774366:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787757042.774386:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757042.774396:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654845440/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757042.774421:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757042.774428:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757042.774432:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757042.774441:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757042/real 1787757042] req@ffff8c05b2d9b480 x1874588654845184/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757042.774449:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10099 00000100:00080000:1.0:1787757042.774464:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757042.774474:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757042.774482:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654845184/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787757042.774484:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787757042.774507:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787757042.774513:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654845568/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757042.774519:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787757042.774566:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787757042.774573:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654845184:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757042.774575:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757042.774582:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757042.774596:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10099 00000100:00080000:1.0:1787757042.774602:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757042.774608:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757042.774616:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757042.774637:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654845312:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757042.774647:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654845696/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757042.774664:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757042.774668:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757042.774672:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757042.774678:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10099 00000100:00080000:1.0:1787757042.774685:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757042.774693:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:0.0:1787757042.774695:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:1.0:1787757042.774704:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757042.774723:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654845824/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757042.774728:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757042/real 1787757042] req@ffff8c05b2d9b100 x1874588654845312/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757042.774755:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757042.774765:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787757042.774768:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757042.774773:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654845312/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757042.774787:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10099 00000100:00080000:0.0:1787757042.774790:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757042.774794:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757042.774801:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787757042.774808:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757042.774816:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654845312:192.168.201.107@tcp:38:kworker.0 00000008:00100000:1.0:1787757042.774820:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757042.774830:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654845440:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757042.774840:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588654845952/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757042.774854:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757042.774858:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757042.774861:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757042.774867:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10099 00000100:00080000:1.0:1787757042.774872:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757042.774920:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757042.774930:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787757042.774955:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757042.774956:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588654846080/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757042.775001:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757042/real 1787757042] req@ffff8c05b2d98380 x1874588654845440/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757042.775021:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654845440/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757042.775034:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757042.775039:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757042.775043:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654845440:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757042.775052:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654845568:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757042.775081:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757042.775094:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757042/real 1787757042] req@ffff8c05b2d99880 x1874588654845568/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757042.775108:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654845568/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757042.775118:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757042.775122:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757042.775126:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654845568:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757042.775148:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654845696:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757042.775168:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757042.775180:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757042/real 1787757042] req@ffff8c05b2d99f80 x1874588654845696/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757042.775194:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654845696/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757042.775204:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757042.775207:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757042.775211:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654845696:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757042.775218:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654845824:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757042.775255:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757042.775266:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757042/real 1787757042] req@ffff8c05b2d98000 x1874588654845824/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757042.775288:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654845824/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757042.775297:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757042.775300:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757042.775303:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654845824:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757042.775309:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654845952:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757042.775326:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757042.775335:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757042/real 1787757042] req@ffff8c05b2d98700 x1874588654845952/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757042.775348:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654845952/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757042.775357:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757042.775360:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757042.775363:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654845952:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757042.775369:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654846080:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757042.775385:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757042.775395:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757042/real 1787757042] req@ffff8c05b2d9a300 x1874588654846080/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757042.775407:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588654846080/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757042.775424:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757042.775428:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757042.775431:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654846080:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757042.902008:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757043.350207:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757043.925943:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757044.374092:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757044.949999:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757045.398226:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757045.974074:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757046.422191:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757046.998059:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757047.446261:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757047.893989:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757047.894002:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787757047.894009:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757047.894025:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10104 00000100:00080000:1.0:1787757047.894043:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757047.894055:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757047.894114:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588654846208/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757047.894159:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757047.894165:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757047.894168:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757047.894174:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10104 00000100:00080000:1.0:1787757047.894180:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757047.894185:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757047.894197:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757047.894218:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588654846336/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757047.894232:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757047.894237:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757047.894240:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757047.894245:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10104 00000100:00080000:1.0:1787757047.894249:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757047.894254:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757047.894259:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757047.894272:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588654846464/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757047.894282:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654846208:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757047.894293:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757047.894299:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757047.894301:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757047.894307:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10104 00000100:00080000:1.0:1787757047.894311:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757047.894316:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757047.894323:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757047.894342:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588654846592/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757047.894355:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757047.894360:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757047.894363:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757047.894366:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10104 00000100:00080000:1.0:1787757047.894371:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787757047.894379:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757047.894384:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757047.894393:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757047.894411:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654846720/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757047.894423:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757047.894428:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757047.894431:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757047.894436:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10104 00000100:00080000:1.0:1787757047.894441:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757047.894479:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757047/real 1787757047] req@ffff8c05b2d9aa00 x1874588654846208/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757047.894497:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757047.894509:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757047.894517:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588654846208/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757047.894533:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654846848/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757047.894539:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757047.894558:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787757047.894561:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787757047.894570:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654846208:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757047.894571:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787757047.894578:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757047.894589:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10104 00000100:00100000:0.0:1787757047.894596:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654846336:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787757047.894599:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757047.894613:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757047.894620:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757047.894639:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654846976/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757047.894654:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757047.894664:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757047.894670:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757047.894673:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757047.894678:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10104 00000100:00080000:1.0:1787757047.894682:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757047.894686:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757047.894693:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757047.894701:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757047/real 1787757047] req@ffff8c05b2d9bb80 x1874588654846336/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757047.894715:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654847104/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757047.894730:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588654846336/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757047.894744:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757047.894749:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757047.894753:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654846336:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757047.894790:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654846464:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757047.894815:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757047.894829:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757047/real 1787757047] req@ffff8c05b2d9a300 x1874588654846464/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757047.894843:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588654846464/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757047.894853:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757047.894856:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757047.894860:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654846464:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757047.894867:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654846592:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757047.894936:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757047.894951:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757047/real 1787757047] req@ffff8c05b2d98700 x1874588654846592/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757047.894966:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654846592/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757047.894977:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757047.894980:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757047.894984:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654846592:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757047.894991:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654846720:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757047.895011:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757047.895023:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757047/real 1787757047] req@ffff8c05b2d98000 x1874588654846720/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757047.895037:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654846720/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757047.895047:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757047.895050:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757047.895054:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654846720:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757047.895060:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654846848:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757047.895079:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757047.895089:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757047/real 1787757047] req@ffff8c05b2d99f80 x1874588654846848/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757047.895103:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654846848/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757047.895113:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757047.895116:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757047.895119:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654846848:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757047.895126:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654846976:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757047.895143:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757047.895153:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757047/real 1787757047] req@ffff8c05b2d99880 x1874588654846976/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757047.895167:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654846976/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757047.895185:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757047.895188:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757047.895192:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654846976:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757047.895199:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654847104:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757047.895216:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757047.895226:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757047/real 1787757047] req@ffff8c05b2d98380 x1874588654847104/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757047.895239:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654847104/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757047.895249:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757047.895252:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757047.895256:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654847104:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757048.022074:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757048.470049:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757049.045964:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757049.494211:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757050.069988:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757050.518124:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757051.093969:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757051.542230:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757052.117986:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757052.566104:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757053.014084:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757053.014102:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787757053.014109:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757053.014126:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10109 00000100:00080000:2.0:1787757053.014154:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757053.014164:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757053.014213:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588654847232/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757053.014251:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757053.014256:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757053.014259:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757053.014264:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10109 00000100:00080000:2.0:1787757053.014270:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757053.014274:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757053.014286:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757053.014302:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588654847360/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757053.014317:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757053.014322:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757053.014325:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757053.014331:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10109 00000100:00080000:2.0:1787757053.014334:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757053.014346:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757053.014349:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654847232:192.168.201.107@tcp:250:kworker.0 00000008:00100000:2.0:1787757053.014377:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757053.014410:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588654847488/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757053.014436:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757053.014444:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757053.014449:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757053.014457:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10109 00000100:00080000:2.0:1787757053.014466:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787757053.014475:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757053.014483:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757053.014494:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757053.014518:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588654847616/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757053.014540:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757053/real 1787757053] req@ffff8c05b2e70000 x1874588654847232/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757053.014547:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757053.014562:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757053.014569:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757053.014584:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10109 00000100:00080000:2.0:1787757053.014590:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757053.014596:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588654847232/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757053.014601:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757053.014614:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787757053.014630:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787757053.014639:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588654847744/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757053.014646:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757053.014664:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654847232:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757053.014667:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757053.014680:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757053.014687:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757053.014693:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10109 00000100:00080000:2.0:1787757053.014697:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757053.014703:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654847360:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757053.014707:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757053.014717:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757053.014738:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588654847872/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757053.014751:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757053.014755:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787757053.014759:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757053.014764:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10109 00000100:00080000:2.0:1787757053.014772:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787757053.014775:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757053.014783:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757053.014792:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757053.014813:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588654848000/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757053.014820:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757053/real 1787757053] req@ffff8c05b2e73100 x1874588654847360/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757053.014857:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757053.014868:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787757053.014925:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588654847360/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757053.014944:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757053.014944:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757053.014962:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10109 00000100:00080000:0.0:1787757053.014964:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787757053.014977:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757053.014983:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654847360:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757053.014991:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757053.015008:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757053.015032:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588654848128/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757053.015051:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654847488:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757053.015101:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757053.015119:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757053/real 1787757053] req@ffff8c05b2e72d80 x1874588654847488/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757053.015135:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588654847488/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757053.015146:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757053.015150:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757053.015154:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654847488:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757053.015162:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654847616:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757053.015183:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757053.015195:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757053/real 1787757053] req@ffff8c05b2e72680 x1874588654847616/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757053.015210:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588654847616/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757053.015220:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757053.015223:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757053.015227:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654847616:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757053.015234:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654847744:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757053.015253:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757053.015264:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757053/real 1787757053] req@ffff8c05b2e70e00 x1874588654847744/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757053.015278:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588654847744/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757053.015287:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757053.015291:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757053.015294:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654847744:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757053.015300:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654847872:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757053.015318:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757053.015329:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757053/real 1787757053] req@ffff8c05b2e72a00 x1874588654847872/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757053.015342:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588654847872/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757053.015393:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757053.015398:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757053.015403:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654847872:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757053.015411:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654848000:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757053.015438:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757053.015452:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757053/real 1787757053] req@ffff8c05b2e71180 x1874588654848000/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757053.015467:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588654848000/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757053.015477:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757053.015481:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757053.015485:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654848000:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757053.015509:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654848128:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757053.015529:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757053.015541:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757053/real 1787757053] req@ffff8c05b2e72300 x1874588654848128/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757053.015555:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588654848128/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757053.015564:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757053.015568:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757053.015573:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654848128:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757053.142008:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757053.590318:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757054.165967:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757054.614051:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757055.189966:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757055.638253:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757056.213963:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757056.662083:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757057.238015:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757057.686234:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757058.134045:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757058.134062:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787757058.134069:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757058.134086:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10114 00000100:00080000:2.0:1787757058.134102:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757058.134114:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757058.134171:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588654848256/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757058.134220:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757058.134229:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757058.134234:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757058.134244:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10114 00000100:00080000:2.0:1787757058.134253:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757058.134264:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757058.134313:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757058.134333:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654848256:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787757058.134343:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588654848384/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757058.134363:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757058.134372:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757058.134376:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757058.134384:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10114 00000100:00080000:2.0:1787757058.134394:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757058.134403:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757058.134413:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757058.134436:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588654848512/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787757058.134446:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757058.134456:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757058.134462:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757058.134465:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757058.134478:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10114 00000100:00080000:2.0:1787757058.134486:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757058.134494:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757058.134505:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757058.134523:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757058/real 1787757058] req@ffff8c05b2e70a80 x1874588654848256/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757058.134525:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588654848640/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757058.134546:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757058.134553:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757058.134557:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787757058.134558:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588654848256/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757058.134577:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757058.134580:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10114 00000100:00080000:1.0:1787757058.134585:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787757058.134587:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787757058.134593:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654848256:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757058.134594:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757058.134604:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757058.134623:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588654848768/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757058.134633:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654848384:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757058.134644:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757058.134650:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757058.134654:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757058.134662:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10114 00000100:00080000:2.0:1787757058.134668:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757058.134675:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:1.0:1787757058.134677:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:2.0:1787757058.134685:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757058.134705:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757058/real 1787757058] req@ffff8c05b2e72300 x1874588654848384/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757058.134721:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588654848896/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757058.134731:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588654848384/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757058.134744:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757058.134749:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757058.134751:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787757058.134757:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757058.134759:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787757058.134765:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10114 00000100:00100000:1.0:1787757058.134767:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654848384:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757058.134773:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787757058.134782:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654848512:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757058.134782:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757058.134792:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757058.134815:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588654849024/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787757058.134822:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757058.134832:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757058.134838:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757058.134842:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787757058.134844:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757058/real 1787757058] req@ffff8c05b2e71180 x1874588654848512/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757058.134849:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10114 00000100:00080000:2.0:1787757058.134855:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757058.134862:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757058.134868:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588654848512/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787757058.134871:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757058.134934:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588654849152/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757058.134947:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757058.134954:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757058.134962:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654848512:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757058.134981:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654848640:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757058.135061:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757058.135083:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757058/real 1787757058] req@ffff8c05b2e72a00 x1874588654848640/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757058.135102:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588654848640/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757058.135114:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757058.135119:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757058.135123:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654848640:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757058.135157:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654848768:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757058.135202:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757058.135226:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757058/real 1787757058] req@ffff8c05b2e70e00 x1874588654848768/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757058.135249:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588654848768/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757058.135291:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757058.135300:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757058.135309:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654848768:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757058.135326:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654848896:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757058.135374:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757058.135397:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757058/real 1787757058] req@ffff8c05b2e72680 x1874588654848896/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757058.135422:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588654848896/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757058.135439:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757058.135443:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757058.135448:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654848896:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757058.135456:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654849024:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757058.135485:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757058.135499:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757058/real 1787757058] req@ffff8c05b2e72d80 x1874588654849024/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757058.135513:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588654849024/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757058.135523:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757058.135527:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757058.135531:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654849024:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757058.135537:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654849152:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757058.135555:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757058.135567:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757058/real 1787757058] req@ffff8c05b2e73100 x1874588654849152/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757058.135581:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588654849152/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757058.135591:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757058.135594:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757058.135597:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654849152:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757058.262012:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757058.710152:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757059.285997:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757059.734204:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757060.310049:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757060.758112:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757061.333991:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757061.782231:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757062.358046:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757062.806111:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757063.254003:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757063.254016:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787757063.254021:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757063.254034:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10119 00000100:00080000:0.0:1787757063.254047:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757063.254056:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757063.254107:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588654849280/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757063.254157:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757063.254165:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757063.254169:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757063.254223:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10119 00000100:00080000:0.0:1787757063.254231:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757063.254237:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757063.254249:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757063.254272:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588654849408/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757063.254295:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757063.254305:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787757063.254308:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654849280:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757063.254311:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757063.254331:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10119 00000100:00080000:0.0:1787757063.254339:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757063.254349:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757063.254360:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757063.254386:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588654849536/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757063.254401:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787757063.254412:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757063.254413:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757063.254433:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757063.254442:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10119 00000100:00080000:0.0:1787757063.254451:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757063.254457:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757063.254465:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757063.254497:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757063/real 1787757063] req@ffff8c05a9f74a80 x1874588654849280/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757063.254498:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588654849664/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757063.254565:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757063.254571:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757063.254579:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757063.254587:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10119 00000100:00080000:0.0:1787757063.254594:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787757063.254600:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588654849280/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757063.254607:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757063.254622:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787757063.254630:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757063.254640:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757063.254647:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588654849792/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757063.254655:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654849280:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757063.254681:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757063.254686:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757063.254689:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757063.254695:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10119 00000100:00080000:0.0:1787757063.254701:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787757063.254705:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654849408:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757063.254711:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757063.254725:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757063.254744:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588654849920/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757063.254758:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757063.254765:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1787757063.254767:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757063.254769:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757063.254781:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10119 00000100:00080000:0.0:1787757063.254787:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787757063.254797:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757063/real 1787757063] req@ffff8c05a9f75c00 x1874588654849408/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757063.254810:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757063.254824:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757063.254849:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588654849408/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757063.254861:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588654850048/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757063.254865:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757063.254919:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757063.254927:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654849408:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757063.254938:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654849536:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757063.254969:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757063.254978:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1787757063.254985:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757063.254988:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757063.255004:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10119 00000100:00100000:1.0:1787757063.255010:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757063/real 1787757063] req@ffff8c05a9f74380 x1874588654849536/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757063.255018:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757063.255039:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757063.255041:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588654849536/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787757063.255053:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787757063.255059:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757063.255068:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757063.255073:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654849536:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757063.255082:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588654850176/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757063.255087:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654849664:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757063.255127:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757063.255144:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757063/real 1787757063] req@ffff8c05a9f76a00 x1874588654849664/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757063.255161:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588654849664/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757063.255216:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757063.255225:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757063.255232:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654849664:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757063.255246:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654849792:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757063.255287:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757063.255310:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757063/real 1787757063] req@ffff8c05a9f76d80 x1874588654849792/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757063.255326:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588654849792/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757063.255337:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757063.255341:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757063.255345:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654849792:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757063.255371:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654849920:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757063.255397:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757063.255413:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757063/real 1787757063] req@ffff8c05a9f77b80 x1874588654849920/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757063.255436:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588654849920/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757063.255454:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757063.255460:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757063.255468:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654849920:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757063.255480:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654850048:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757063.255517:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757063.255534:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757063/real 1787757063] req@ffff8c05a9f76680 x1874588654850048/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757063.255549:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588654850048/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757063.255559:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757063.255563:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757063.255567:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654850048:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757063.255574:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654850176:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757063.255595:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757063.255607:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757063/real 1787757063] req@ffff8c05a9f74a80 x1874588654850176/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757063.255621:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588654850176/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757063.255630:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757063.255633:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757063.255637:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654850176:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757063.381978:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757063.830251:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757064.406047:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757064.854197:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757065.429998:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757065.878319:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757066.454051:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757066.902138:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757067.478065:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757067.926376:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757068.374119:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757068.374133:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787757068.374138:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757068.374151:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10124 00000100:00080000:1.0:1787757068.374165:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757068.374174:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757068.374224:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588654850304/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757068.374265:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757068.374270:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757068.374272:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757068.374277:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10124 00000100:00080000:1.0:1787757068.374281:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757068.374286:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757068.374299:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757068.374326:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588654850432/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757068.374337:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654850304:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757068.374349:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757068.374356:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757068.374359:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757068.374365:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10124 00000100:00080000:1.0:1787757068.374371:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757068.374376:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757068.374384:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757068.374403:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654850560/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757068.374417:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757068.374422:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757068.374424:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757068.374430:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10124 00000100:00080000:1.0:1787757068.374434:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757068.374439:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757068.374445:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757068.374462:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654850688/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757068.374474:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757068.374479:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757068.374482:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757068.374497:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654850432:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787757068.374497:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10124 00000100:00080000:1.0:1787757068.374503:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757068.374513:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757068.374522:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757068.374541:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654850816/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757068.374556:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787757068.374558:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654850560:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787757068.374563:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757068.374573:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757068.374579:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10124 00000100:00080000:1.0:1787757068.374587:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757068.374596:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757068.374609:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757068.374626:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654850688:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757068.374632:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654850944/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757068.374647:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757068.374653:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787757068.374657:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757068.374664:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10124 00000100:00080000:1.0:1787757068.374676:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757068.374682:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757068.374688:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757068.374692:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654850816:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757068.374711:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588654851072/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757068.374727:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787757068.374730:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654850944:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787757068.374736:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757068.374742:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757068.374749:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10124 00000100:00080000:1.0:1787757068.374766:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757068.374773:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757068.374780:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654851072:192.168.201.107@tcp:8:kworker.0 00000008:00100000:1.0:1787757068.374785:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757068.374812:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588654851200/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757068.374863:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654851200:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757068.502041:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757068.950161:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757069.525973:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757069.974211:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757070.549965:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757070.998063:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787757071.446304:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.107:988 00000400:00000100:2.0:1787757071.446376:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.107@tcp at host 192.168.201.107 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787757071.446399:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787757071.446447:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787757071.446459:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787757071.446469:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787757071.446477:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787757071.446487:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787757071.446496:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787757071.446504:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:0.0:1787757071.446601:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757068/real 1787757071] req@ffff8c05b2d9a300 x1874588654851200/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757071.446695:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588654851200/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757071.446720:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757071.446730:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757071.446740:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654851200:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757071.446758:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757068/real 1787757071] req@ffff8c05b2d98700 x1874588654851072/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757071.446785:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654851072/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757071.446801:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757071.446808:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757071.446815:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654851072:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757071.446831:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757068/real 1787757071] req@ffff8c05b2d99f80 x1874588654850816/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757071.446938:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654850816/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757071.446958:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757071.446965:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757071.446972:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654850816:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757071.446986:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757068/real 1787757071] req@ffff8c05b2d98000 x1874588654850944/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757071.447011:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654850944/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757071.447025:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757071.447064:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757071.447071:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654850944:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757071.447094:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757068/real 1787757071] req@ffff8c05b2d9b480 x1874588654850432/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757071.447129:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654850432/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757071.447156:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757071.447164:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757071.447172:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654850432:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757071.447187:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757068/real 1787757071] req@ffff8c05b2d98380 x1874588654850560/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757071.447213:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654850560/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757071.447229:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757071.447237:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757071.447243:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654850560:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757071.447258:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757068/real 1787757071] req@ffff8c05b2d99880 x1874588654850688/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757071.447283:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654850688/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757071.447300:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757071.447307:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757071.447313:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654850688:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757071.447328:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757068/real 1787757071] req@ffff8c05b2d9b100 x1874588654850304/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757071.447353:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654850304/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757071.447367:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757071.447372:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757071.447378:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654850304:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787757071.574023:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757072.022308:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757072.598050:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757073.046153:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757073.494108:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757073.494125:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787757073.494133:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757073.494149:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10129 00000100:00080000:1.0:1787757073.494167:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757073.494179:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757073.494242:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588654851328/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757073.494291:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757073.494316:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757073.494320:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757073.494331:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10129 00000100:00080000:1.0:1787757073.494340:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757073.494351:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757073.494371:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757073.494386:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654851328:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787757073.494406:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588654851456/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757073.494442:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757073.494451:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757073.494456:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757073.494464:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10129 00000100:00080000:1.0:1787757073.494503:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757073.494511:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757073.494522:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787757073.494528:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757073.494558:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588654851584/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757073.494582:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757073.494589:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757073.494603:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757073.494612:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10129 00000100:00080000:1.0:1787757073.494620:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757073.494630:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757073.494642:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757073.494648:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757073/real 1787757073] req@ffff8c05b2d9bb80 x1874588654851328/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757073.494676:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654851712/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757073.494701:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588654851328/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757073.494703:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757073.494722:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757073.494731:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757073.494751:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10129 00000100:00080000:0.0:1787757073.494753:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757073.494764:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757073.494771:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787757073.494784:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757073.494794:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654851328:192.168.201.107@tcp:250:kworker.0 00000008:00100000:1.0:1787757073.494802:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757073.494836:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654851840/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757073.494856:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757073.494863:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757073.494868:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757073.494944:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654851456:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787757073.494959:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10129 00000100:00080000:1.0:1787757073.494971:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757073.494981:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757073.494994:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787757073.495049:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757073.495061:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588654851968/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757073.495090:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757073.495099:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787757073.495104:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757073.495117:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757073/real 1787757073] req@ffff8c05b2d9aa00 x1874588654851456/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757073.495119:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10129 00000100:00080000:1.0:1787757073.495133:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757073.495155:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757073.495173:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757073.495185:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588654851456/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757073.495204:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654852096/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757073.495211:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757073.495233:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787757073.495246:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787757073.495248:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654851456:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787757073.495263:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757073.495271:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757073.495277:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654851584:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787757073.495287:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10129 00000100:00080000:1.0:1787757073.495301:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757073.495310:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757073.495321:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757073.495346:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654852224/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757073.495368:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757073.495426:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757073/real 1787757073] req@ffff8c05b2d9b100 x1874588654851584/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757073.495452:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654851584/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757073.495469:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757073.495475:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757073.495481:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654851584:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757073.495493:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654851712:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757073.495531:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757073.495552:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757073/real 1787757073] req@ffff8c05b2d99880 x1874588654851712/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757073.495573:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654851712/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757073.495588:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757073.495595:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757073.495602:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654851712:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757073.495614:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654851840:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757073.495665:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757073.495686:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757073/real 1787757073] req@ffff8c05b2d98380 x1874588654851840/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757073.495708:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654851840/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757073.495721:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757073.495727:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757073.495732:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654851840:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757073.495771:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654851968:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757073.495807:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757073.495825:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757073/real 1787757073] req@ffff8c05b2d9b480 x1874588654851968/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757073.495844:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654851968/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757073.495858:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757073.495864:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757073.495869:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654851968:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757073.495933:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654852096:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757073.495969:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757073.495992:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757073/real 1787757073] req@ffff8c05b2d98000 x1874588654852096/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757073.496054:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654852096/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757073.496068:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757073.496074:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757073.496079:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654852096:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757073.496091:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654852224:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757073.496126:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757073.496145:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757073/real 1787757073] req@ffff8c05b2d99f80 x1874588654852224/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757073.496165:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654852224/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757073.496179:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757073.496185:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757073.496191:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654852224:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757073.622059:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757074.070227:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757074.646037:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757075.094124:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757075.670027:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757076.118270:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757076.694046:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757077.142137:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757077.718059:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757078.166279:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757078.614073:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757078.614085:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787757078.614090:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757078.614101:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10135 00000100:00080000:0.0:1787757078.614113:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757078.614120:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757078.614175:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588654852352/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757078.614210:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757078.614214:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757078.614216:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757078.614220:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10135 00000100:00080000:0.0:1787757078.614224:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757078.614229:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757078.614239:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757078.614257:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588654852480/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757078.614260:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654852352:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757078.614282:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757078.614286:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757078.614288:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757078.614293:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10135 00000100:00080000:0.0:1787757078.614296:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757078.614300:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757078.614311:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757078.614326:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588654852608/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757078.614336:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757078.614339:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757078.614342:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757078.614346:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10135 00000100:00080000:0.0:1787757078.614351:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757078.614360:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:1.0:1787757078.614360:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:0.0:1787757078.614372:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757078.614386:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588654852736/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757078.614396:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757078.614401:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757078.614403:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757078.614407:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10135 00000100:00080000:0.0:1787757078.614410:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757078.614418:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757078.614420:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757078/real 1787757078] req@ffff8c05a9f77800 x1874588654852352/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787757078.614423:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757078.614463:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588654852864/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757078.614477:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588654852352/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757078.614481:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757078.614497:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757078.614504:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757078.614513:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757078.614515:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10135 00000100:00080000:0.0:1787757078.614528:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757078.614529:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787757078.614539:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757078.614544:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654852352:192.168.201.107@tcp:250:kworker.0 00000008:00100000:0.0:1787757078.614550:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757078.614583:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588654852992/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757078.614596:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654852480:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757078.614614:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757078.614641:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787757078.614644:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787757078.614652:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757078.614653:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10135 00000100:00080000:0.0:1787757078.614664:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757078.614670:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757078.614677:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757078.614683:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757078/real 1787757078] req@ffff8c05a9f76300 x1874588654852480/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757078.614697:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588654853120/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757078.614707:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588654852480/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757078.614716:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757078.614734:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757078.614738:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757078.614745:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757078.614747:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10135 00000100:00080000:0.0:1787757078.614754:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757078.614755:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787757078.614764:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757078.614768:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654852480:192.168.201.107@tcp:38:kworker.0 00000008:00100000:0.0:1787757078.614772:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757078.614779:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654852608:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757078.614791:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588654853248/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787757078.614807:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757078.614822:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757078/real 1787757078] req@ffff8c05a9f74e00 x1874588654852608/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757078.614835:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588654852608/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757078.614844:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757078.614847:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757078.614850:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654852608:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757078.614856:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654852736:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757078.614872:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757078.614929:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757078/real 1787757078] req@ffff8c05a9f74000 x1874588654852736/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757078.614945:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588654852736/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757078.614953:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757078.614956:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757078.614960:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654852736:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757078.614966:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654852864:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757078.614986:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757078.614998:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757078/real 1787757078] req@ffff8c05a9f75880 x1874588654852864/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757078.615009:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588654852864/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757078.615017:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757078.615019:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757078.615022:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654852864:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757078.615039:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654852992:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757078.615055:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757078.615065:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757078/real 1787757078] req@ffff8c05a9f75500 x1874588654852992/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757078.615076:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588654852992/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757078.615084:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757078.615087:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757078.615089:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654852992:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757078.615095:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654853120:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757078.615109:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757078.615119:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757078/real 1787757078] req@ffff8c05a9f77100 x1874588654853120/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757078.615130:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588654853120/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757078.615138:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757078.615140:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757078.615143:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654853120:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757078.615148:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654853248:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757078.615162:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757078.615170:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757078/real 1787757078] req@ffff8c05a9f75180 x1874588654853248/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757078.615181:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588654853248/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757078.615189:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757078.615192:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757078.615195:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654853248:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757078.741963:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757079.190109:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757079.766025:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757080.214234:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757080.789978:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757081.238126:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757081.814100:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757082.262245:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757082.838013:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757083.286143:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757083.734122:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757083.734132:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787757083.734136:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757083.734146:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10140 00000100:00080000:1.0:1787757083.734181:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757083.734189:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757083.734248:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588654853376/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757083.734282:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757083.734287:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757083.734289:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757083.734311:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10140 00000100:00080000:1.0:1787757083.734315:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757083.734322:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757083.734332:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757083.734347:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588654853504/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757083.734383:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757083.734393:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787757083.734397:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654853376:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757083.734398:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757083.734409:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10140 00000100:00080000:1.0:1787757083.734422:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757083.734428:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757083.734432:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757083.734446:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654853632/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757083.734466:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757083.734469:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757083.734471:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757083.734476:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10140 00000100:00080000:1.0:1787757083.734481:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757083.734486:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757083.734492:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757083.734504:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654853760/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757083.734511:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757083.734513:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757083.734514:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757083.734517:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10140 00000100:00080000:1.0:1787757083.734530:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757083.734533:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757083.734537:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757083.734549:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588654853888/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757083.734555:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757083.734566:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757083.734569:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757083.734571:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757083.734575:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10140 00000100:00080000:1.0:1787757083.734583:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757083.734587:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757083.734591:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757083.734601:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654854016/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757083.734608:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757083.734611:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787757083.734612:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757083.734616:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10140 00000100:00080000:1.0:1787757083.734618:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757083.734621:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757083.734624:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757083.734636:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654854144/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757083.734644:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757083.734648:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757083.734650:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757083.734660:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10140 00000100:00100000:0.0:1787757083.734662:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757083/real 1787757083] req@ffff8c05b2d98700 x1874588654853376/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757083.734671:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757083.734690:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757083.734702:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757083.734731:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588654854272/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757083.734745:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654853376/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757083.734776:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757083.734784:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757083.734791:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654853376:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787757083.734862:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654853504:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757083.734933:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757083.734953:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757083/real 1787757083] req@ffff8c05b2d9a300 x1874588654853504/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757083.734971:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588654853504/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757083.734982:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757083.734987:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757083.734991:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654853504:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757083.735000:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654853632:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757083.735022:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757083.735035:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757083/real 1787757083] req@ffff8c05b2d99f80 x1874588654853632/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757083.735049:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654853632/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757083.735067:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757083.735071:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757083.735075:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654853632:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757083.735082:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654853760:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757083.735103:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757083.735114:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757083/real 1787757083] req@ffff8c05b2d98000 x1874588654853760/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757083.735128:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654853760/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757083.735139:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757083.735143:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757083.735146:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654853760:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757083.735153:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654853888:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757083.735174:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757083.735186:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757083/real 1787757083] req@ffff8c05b2d9b480 x1874588654853888/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757083.735199:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654853888/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757083.735209:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757083.735213:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757083.735218:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654853888:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757083.735224:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654854016:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757083.735244:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757083.735255:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757083/real 1787757083] req@ffff8c05b2d98380 x1874588654854016/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757083.735269:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654854016/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757083.735279:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757083.735282:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757083.735286:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654854016:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757083.735292:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654854144:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757083.735311:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757083.735322:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757083/real 1787757083] req@ffff8c05b2d99880 x1874588654854144/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757083.735335:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654854144/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757083.735345:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757083.735349:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757083.735353:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654854144:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757083.735359:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654854272:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757083.735377:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757083.735389:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757083/real 1787757083] req@ffff8c05b2d9b100 x1874588654854272/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757083.735403:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654854272/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757083.735413:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757083.735417:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757083.735420:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654854272:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757083.861965:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757084.310201:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757084.886076:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757085.334111:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757085.910063:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757086.358206:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757086.934072:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757087.382144:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757087.958023:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757088.406269:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757088.853985:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757088.853997:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787757088.854002:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757088.854014:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10145 00000100:00080000:1.0:1787757088.854027:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757088.854036:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757088.854084:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588654854400/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757088.854133:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757088.854138:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757088.854141:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757088.854146:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10145 00000100:00080000:1.0:1787757088.854150:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757088.854155:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757088.854165:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757088.854183:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588654854528/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757088.854199:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757088.854204:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757088.854210:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757088.854213:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654854400:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757088.854222:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10145 00000100:00080000:1.0:1787757088.854230:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757088.854234:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757088.854241:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757088.854260:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588654854656/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757088.854276:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757088.854281:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757088.854283:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757088.854288:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10145 00000100:00080000:1.0:1787757088.854293:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757088.854297:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757088.854303:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787757088.854323:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757088.854324:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654854784/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757088.854343:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757088.854348:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757088.854351:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757088.854358:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10145 00000100:00080000:1.0:1787757088.854362:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757088.854366:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757088.854372:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757088.854399:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757088/real 1787757088] req@ffff8c05b2d9aa00 x1874588654854400/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757088.854401:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654854912/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757088.854451:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588654854400/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757088.854452:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757088.854470:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757088.854476:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757088.854484:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757088.854487:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10145 00000100:00080000:1.0:1787757088.854497:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757088.854498:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787757088.854509:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757088.854512:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654854400:192.168.201.107@tcp:250:kworker.0 00000008:00100000:1.0:1787757088.854519:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757088.854539:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588654855040/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757088.854554:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654854528:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787757088.854555:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757088.854571:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787757088.854574:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757088.854581:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10145 00000100:00080000:1.0:1787757088.854588:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757088.854593:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757088.854600:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787757088.854607:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757088.854621:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654855168/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757088.854635:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757088/real 1787757088] req@ffff8c05b2d9bb80 x1874588654854528/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757088.854644:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757088.854658:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757088.854665:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757088.854676:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10145 00000100:00100000:0.0:1787757088.854682:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588654854528/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757088.854685:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757088.854700:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757088.854711:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757088.854747:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654855296/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757088.854782:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757088.854793:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757088.854800:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654854528:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757088.854812:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654854656:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757088.854846:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757088.854862:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757088/real 1787757088] req@ffff8c05b2d9b100 x1874588654854656/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757088.854920:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654854656/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757088.854932:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757088.854936:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757088.854940:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654854656:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757088.854960:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654854784:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757088.854988:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757088.855007:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757088/real 1787757088] req@ffff8c05b2d99880 x1874588654854784/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757088.855046:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654854784/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757088.855064:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757088.855070:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757088.855078:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654854784:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757088.855091:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654854912:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757088.855130:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757088.855147:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757088/real 1787757088] req@ffff8c05b2d98380 x1874588654854912/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757088.855162:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654854912/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757088.855173:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757088.855177:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757088.855181:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654854912:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757088.855204:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654855040:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757088.855226:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757088.855239:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757088/real 1787757088] req@ffff8c05b2d9b480 x1874588654855040/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757088.855253:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654855040/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757088.855262:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757088.855266:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757088.855269:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654855040:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757088.855276:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654855168:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757088.855294:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757088.855306:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757088/real 1787757088] req@ffff8c05b2d98000 x1874588654855168/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757088.855320:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654855168/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757088.855330:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757088.855333:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757088.855337:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654855168:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757088.855343:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654855296:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757088.855360:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757088.855371:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757088/real 1787757088] req@ffff8c05b2d99f80 x1874588654855296/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757088.855385:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654855296/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757088.855394:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757088.855397:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757088.855401:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654855296:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757088.982076:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757089.430197:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757090.006020:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757090.454335:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757091.030013:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757091.478150:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757092.053993:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757092.502216:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757093.077929:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757093.526103:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757093.973965:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757093.973977:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787757093.973982:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757093.973994:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10150 00000100:00080000:1.0:1787757093.974008:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757093.974016:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757093.974065:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588654855424/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757093.974108:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757093.974112:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757093.974115:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757093.974122:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10150 00000100:00080000:1.0:1787757093.974125:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757093.974130:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757093.974141:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757093.974157:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588654855552/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757093.974158:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654855424:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757093.974173:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757093.974178:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757093.974179:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757093.974183:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10150 00000100:00080000:1.0:1787757093.974186:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757093.974190:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757093.974194:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757093.974203:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654855680/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757093.974212:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757093.974215:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757093.974216:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757093.974219:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10150 00000100:00080000:1.0:1787757093.974221:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757093.974224:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757093.974231:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757093.974248:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654855808/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787757093.974257:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757093.974258:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757093.974263:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757093.974266:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757093.974273:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10150 00000100:00080000:1.0:1787757093.974282:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757093.974290:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757093.974297:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757093.974312:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588654855936/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757093.974327:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757093.974332:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757093.974335:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757093.974343:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10150 00000100:00100000:2.0:1787757093.974344:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757093/real 1787757093] req@ffff8c05b2d9a300 x1874588654855424/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757093.974348:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757093.974356:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757093.974363:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757093.974370:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588654855424/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757093.974378:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654856064/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757093.974384:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757093.974387:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757093.974392:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787757093.974395:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787757093.974399:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654855424:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757093.974401:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757093.974412:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10150 00000100:00080000:1.0:1787757093.974420:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757093.974426:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757093.974432:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757093.974444:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654856192/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757093.974448:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654855552:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787757093.974464:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757093.974467:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757093.974469:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757093.974472:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10150 00000800:00000100:2.0:1787757093.974477:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757093.974479:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757093.974485:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757093.974488:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757093.974493:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757093/real 1787757093] req@ffff8c05b2d98700 x1874588654855552/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757093.974501:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588654856320/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757093.974509:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654855552/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757093.974522:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757093.974526:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757093.974529:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654855552:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757093.974534:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654855680:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757093.974550:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757093.974557:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757093/real 1787757093] req@ffff8c05b2d99f80 x1874588654855680/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757093.974566:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654855680/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757093.974573:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757093.974576:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757093.974578:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654855680:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757093.974582:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654855808:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757093.974594:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757093.974599:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757093/real 1787757093] req@ffff8c05b2d98000 x1874588654855808/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757093.974606:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654855808/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757093.974611:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757093.974613:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757093.974616:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654855808:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757093.974619:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654855936:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757093.974631:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757093.974653:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757093/real 1787757093] req@ffff8c05b2d9b480 x1874588654855936/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757093.974661:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654855936/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757093.974666:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757093.974668:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757093.974671:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654855936:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757093.974674:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654856064:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757093.974689:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757093.974697:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757093/real 1787757093] req@ffff8c05b2d98380 x1874588654856064/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757093.974705:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654856064/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757093.974710:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757093.974712:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757093.974715:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654856064:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757093.974729:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654856192:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757093.974741:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757093.974750:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757093/real 1787757093] req@ffff8c05b2d99880 x1874588654856192/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757093.974759:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654856192/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757093.974766:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757093.974769:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757093.974772:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654856192:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757093.974777:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654856320:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757093.974789:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757093.974797:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757093/real 1787757093] req@ffff8c05b2d9b100 x1874588654856320/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757093.974808:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654856320/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757093.974814:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757093.974817:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757093.974819:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654856320:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757094.101961:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757094.550215:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757095.125998:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757095.574146:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757096.150021:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757096.598195:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757097.174007:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757097.622135:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757098.198045:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757098.646285:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757099.094030:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757099.094048:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787757099.094056:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757099.094072:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10155 00000100:00080000:0.0:1787757099.094089:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757099.094098:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757099.094149:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588654856448/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757099.094197:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757099.094205:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757099.094209:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757099.094218:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10155 00000100:00080000:0.0:1787757099.094229:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757099.094239:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757099.094257:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757099.094267:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654856448:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787757099.094295:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588654856576/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757099.094323:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757099.094331:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757099.094335:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757099.094344:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10155 00000800:00000100:2.0:1787757099.094360:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757099.094369:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757099.094380:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757099.094390:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757099.094436:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588654856704/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757099.094448:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757099/real 1787757099] req@ffff8c05a9f75180 x1874588654856448/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757099.094458:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757099.094470:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757099.094494:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757099.094505:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10155 00000100:00080000:0.0:1787757099.094511:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787757099.094517:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588654856448/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757099.094525:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757099.094542:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787757099.094576:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757099.094597:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757099.094606:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588654856832/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757099.094609:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654856448:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757099.094642:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757099.094650:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1787757099.094657:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654856576:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757099.094659:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757099.094686:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10155 00000100:00080000:0.0:1787757099.094696:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757099.094704:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757099.094713:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787757099.094725:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757099.094738:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588654856960/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757099.094757:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787757099.094764:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757099/real 1787757099] req@ffff8c05a9f77100 x1874588654856576/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757099.094765:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757099.094772:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757099.094785:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10155 00000100:00080000:0.0:1787757099.094797:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757099.094807:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757099.094815:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757099.094823:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588654856576/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757099.094835:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588654857088/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757099.094845:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757099.094858:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787757099.094869:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787757099.094871:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654856576:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757099.094930:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787757099.094938:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757099.094953:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10155 00000100:00100000:2.0:1787757099.094959:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654856704:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757099.094963:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757099.094976:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757099.094990:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757099.095023:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588654857216/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787757099.095034:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757099.095057:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787757099.095066:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757099/real 1787757099] req@ffff8c05a9f75500 x1874588654856704/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757099.095069:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757099.095079:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757099.095094:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10155 00000100:00100000:2.0:1787757099.095103:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588654856704/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757099.095104:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757099.095121:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757099.095136:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787757099.095143:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757099.095152:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757099.095158:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654856704:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757099.095169:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588654857344/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757099.095177:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654856832:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757099.095225:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757099.095244:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757099/real 1787757099] req@ffff8c05a9f75880 x1874588654856832/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757099.095261:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588654856832/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757099.095273:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757099.095279:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757099.095285:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654856832:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757099.095318:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654856960:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757099.095359:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757099.095383:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757099/real 1787757099] req@ffff8c05a9f74000 x1874588654856960/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757099.095407:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588654856960/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757099.095418:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757099.095422:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757099.095426:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654856960:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757099.095434:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654857088:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757099.095461:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757099.095474:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757099/real 1787757099] req@ffff8c05a9f74e00 x1874588654857088/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757099.095489:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588654857088/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757099.095498:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757099.095502:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757099.095505:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654857088:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757099.095512:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654857216:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757099.095530:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757099.095541:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757099/real 1787757099] req@ffff8c05a9f76300 x1874588654857216/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757099.095600:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588654857216/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757099.095611:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757099.095617:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757099.095624:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654857216:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757099.095636:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654857344:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757099.095676:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757099.095701:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757099/real 1787757099] req@ffff8c05a9f77800 x1874588654857344/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757099.095727:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588654857344/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757099.095743:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757099.095749:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757099.095756:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654857344:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757099.222027:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757099.670118:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757100.246008:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757100.694246:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757101.270039:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757101.718148:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757102.294068:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757102.742197:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757103.318060:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757103.766139:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757104.214025:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757104.214038:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757104.214044:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757104.214081:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10160 00000100:00080000:3.0:1787757104.214096:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757104.214105:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757104.214163:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588654857472/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757104.214225:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757104.214249:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757104.214253:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757104.214258:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10160 00000100:00080000:3.0:1787757104.214264:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757104.214269:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757104.214282:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757104.214303:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588654857600/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757104.214338:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787757104.214341:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654857472:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757104.214346:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757104.214354:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757104.214362:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10160 00000100:00080000:3.0:1787757104.214366:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757104.214371:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757104.214378:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757104.214394:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588654857728/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757104.214407:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757104.214412:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757104.214415:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757104.214421:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10160 00000100:00080000:3.0:1787757104.214437:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757104.214443:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757104.214450:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787757104.214482:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757104.214515:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588654857856/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757104.214536:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757104.214543:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757104.214547:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757104.214554:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10160 00000100:00080000:3.0:1787757104.214563:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787757104.214571:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757104/real 1787757104] req@ffff8c05a9f79500 x1874588654857472/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757104.214579:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757104.214598:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757104.214613:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588654857472/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757104.214626:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588654857984/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757104.214635:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757104.214647:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757104.214649:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787757104.214657:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787757104.214664:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654857472:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757104.214665:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757104.214680:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10160 00000100:00080000:3.0:1787757104.214687:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757104.214693:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757104.214698:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757104.214709:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654857600:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757104.214716:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588654858112/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757104.214732:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757104.214737:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757104.214739:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757104.214744:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10160 00000100:00080000:3.0:1787757104.214750:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787757104.214755:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757104.214762:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757104.214771:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757104.214785:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757104/real 1787757104] req@ffff8c05a9f7b100 x1874588654857600/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757104.214791:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588654858240/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757104.214839:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757104.214848:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787757104.214849:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588654857600/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757104.214854:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757104.214867:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10160 00000100:00080000:3.0:1787757104.214875:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757104.214879:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757104.214885:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787757104.214932:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757104.214939:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757104.214945:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654857600:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757104.214955:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654857728:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757104.214968:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588654858368/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787757104.215019:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757104.215044:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757104/real 1787757104] req@ffff8c05a9f7b800 x1874588654857728/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757104.215065:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588654857728/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757104.215076:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757104.215081:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757104.215085:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654857728:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757104.215092:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654857856:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757104.215112:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757104.215124:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757104/real 1787757104] req@ffff8c05a9f7bb80 x1874588654857856/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757104.215138:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588654857856/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757104.215149:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757104.215152:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757104.215156:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654857856:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757104.215162:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654857984:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757104.215180:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757104.215191:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757104/real 1787757104] req@ffff8c05a9f7ad80 x1874588654857984/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757104.215204:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588654857984/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757104.215214:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757104.215218:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757104.215221:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654857984:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757104.215260:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654858112:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757104.215280:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757104.215291:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757104/real 1787757104] req@ffff8c05a9f79f80 x1874588654858112/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757104.215303:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588654858112/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757104.215312:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757104.215316:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757104.215320:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654858112:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757104.215326:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654858240:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757104.215365:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757104.215376:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757104/real 1787757104] req@ffff8c05a9f79180 x1874588654858240/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757104.215389:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588654858240/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757104.215400:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757104.215403:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757104.215406:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654858240:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757104.215412:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654858368:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757104.215429:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757104.215440:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757104/real 1787757104] req@ffff8c05a9f78e00 x1874588654858368/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757104.215497:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588654858368/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757104.215508:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757104.215512:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757104.215516:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654858368:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757104.342047:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757104.790240:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757105.366015:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757105.814145:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757106.390027:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757106.838216:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757107.414030:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757107.862162:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757108.438034:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757108.886195:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757109.334052:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757109.334096:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787757109.334103:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757109.334119:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10165 00000100:00080000:1.0:1787757109.334139:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757109.334150:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757109.334245:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588654858496/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757109.334311:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757109.334320:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757109.334324:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757109.334333:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10165 00000100:00080000:1.0:1787757109.334342:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757109.334353:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757109.334424:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757109.334451:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588654858624/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757109.334453:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654858496:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757109.334477:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757109.334482:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757109.334484:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757109.334489:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10165 00000100:00080000:1.0:1787757109.334493:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757109.334497:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757109.334503:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757109.334520:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588654858752/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757109.334531:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757109.334535:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757109.334537:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757109.334543:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10165 00000100:00080000:1.0:1787757109.334547:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757109.334551:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757109.334557:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787757109.334562:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757109.334574:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654858880/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757109.334587:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757109.334592:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757109.334594:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757109.334597:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10165 00000100:00080000:1.0:1787757109.334601:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757109.334605:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757109.334610:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757109.334622:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654859008/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757109.334630:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757109/real 1787757109] req@ffff8c05b2d9bb80 x1874588654858496/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757109.334635:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757109.334648:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757109.334653:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757109.334662:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10165 00000100:00080000:1.0:1787757109.334666:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787757109.334672:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588654858496/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757109.334674:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757109.334684:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787757109.334694:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787757109.334700:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588654859136/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757109.334704:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757109.334713:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654858496:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757109.334728:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757109.334732:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787757109.334734:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757109.334740:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10165 00000100:00080000:1.0:1787757109.334744:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787757109.334747:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654858624:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787757109.334753:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757109.334760:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757109.334777:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654859264/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757109.334789:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757109.334792:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757109.334794:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787757109.334799:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757109.334806:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10165 00000100:00080000:1.0:1787757109.334814:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787757109.334820:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757109/real 1787757109] req@ffff8c05b2d9aa00 x1874588654858624/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757109.334827:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757109.334843:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757109.334846:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588654858624/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757109.334860:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757109.334864:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757109.334871:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654859392/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757109.334937:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654858624:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757109.334960:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654858752:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757109.335002:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757109.335015:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757109/real 1787757109] req@ffff8c05b2d9b100 x1874588654858752/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757109.335027:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654858752/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757109.335064:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757109.335068:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757109.335072:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654858752:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757109.335080:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654858880:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757109.335121:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757109.335132:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757109/real 1787757109] req@ffff8c05b2d99880 x1874588654858880/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757109.335146:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654858880/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757109.335156:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757109.335159:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757109.335162:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654858880:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757109.335184:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654859008:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757109.335206:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757109.335218:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757109/real 1787757109] req@ffff8c05b2d98380 x1874588654859008/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757109.335242:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654859008/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757109.335252:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757109.335255:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757109.335259:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654859008:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757109.335265:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654859136:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757109.335311:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757109.335332:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757109/real 1787757109] req@ffff8c05b2d9b480 x1874588654859136/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757109.335358:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654859136/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757109.335460:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757109.335464:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757109.335467:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654859136:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757109.335489:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654859264:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757109.335513:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757109.335524:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757109/real 1787757109] req@ffff8c05b2d98000 x1874588654859264/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757109.335535:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654859264/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757109.335550:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757109.335553:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757109.335556:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654859264:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757109.335561:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654859392:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757109.335576:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757109.335585:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757109/real 1787757109] req@ffff8c05b2d99f80 x1874588654859392/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757109.335595:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654859392/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757109.335603:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757109.335606:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757109.335609:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654859392:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757109.462070:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757109.910190:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757110.486091:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757110.934282:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757111.509995:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757111.958098:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757112.534030:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757112.982198:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757113.558019:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757114.006153:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757114.453936:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757114.453944:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787757114.453948:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757114.453956:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10170 00000100:00080000:1.0:1787757114.453965:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757114.453971:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757114.454001:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588654859520/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757114.454025:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757114.454028:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757114.454029:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757114.454032:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10170 00000100:00080000:1.0:1787757114.454034:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757114.454036:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757114.454044:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757114.454053:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588654859648/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757114.454062:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787757114.454064:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654859520:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757114.454067:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757114.454071:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757114.454075:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10170 00000100:00080000:1.0:1787757114.454077:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757114.454080:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757114.454083:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757114.454092:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654859776/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757114.454108:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757114.454111:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757114.454113:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757114.454118:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10170 00000100:00080000:1.0:1787757114.454122:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787757114.454125:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757114.454130:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757114.454137:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757114.454152:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654859904/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757114.454164:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757114/real 1787757114] req@ffff8c05b2d98700 x1874588654859520/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757114.454165:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757114.454174:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757114.454178:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757114.454185:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10170 00000100:00100000:2.0:1787757114.454190:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654859520/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757114.454191:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757114.454201:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757114.454211:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787757114.454213:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757114.454220:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757114.454227:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588654860032/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757114.454228:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654859520:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757114.454245:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757114.454248:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757114.454250:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757114.454255:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10170 00000100:00100000:2.0:1787757114.454258:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654859648:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787757114.454262:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757114.454268:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757114.454293:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787757114.454298:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757114.454310:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654860160/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757114.454313:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757114/real 1787757114] req@ffff8c05b2d9a300 x1874588654859648/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757114.454344:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588654859648/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757114.454345:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757114.454356:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787757114.454360:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757114.454361:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757114.454368:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10170 00000100:00080000:2.0:1787757114.454369:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757114.454375:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654859648:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787757114.454377:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787757114.454382:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654859776:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787757114.454386:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757114.454401:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787757114.454402:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757114.454412:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757114/real 1787757114] req@ffff8c05b2d99f80 x1874588654859776/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757114.454420:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654860288/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757114.454426:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654859776/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757114.454444:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757114.454444:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757114.454451:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787757114.454453:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787757114.454456:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654859776:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787757114.454458:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757114.454466:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10170 00000100:00100000:2.0:1787757114.454467:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654859904:192.168.201.107@tcp:8:kworker.0 00000100:00080000:1.0:1787757114.454472:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757114.454479:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757114.454485:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787757114.454486:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757114.454497:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757114/real 1787757114] req@ffff8c05b2d98000 x1874588654859904/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757114.454502:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588654860416/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757114.454510:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654859904/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757114.454531:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757114.454538:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757114.454541:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654859904:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757114.454546:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654860032:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757114.454560:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757114.454567:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757114/real 1787757114] req@ffff8c05b2d9b480 x1874588654860032/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757114.454574:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654860032/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757114.454579:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757114.454581:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757114.454583:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654860032:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757114.454595:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654860160:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757114.454606:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757114.454612:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757114/real 1787757114] req@ffff8c05b2d98380 x1874588654860160/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757114.454620:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654860160/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757114.454625:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757114.454627:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757114.454629:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654860160:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757114.454632:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654860288:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757114.454642:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757114.454656:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757114/real 1787757114] req@ffff8c05b2d99880 x1874588654860288/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757114.454664:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654860288/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757114.454669:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757114.454671:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757114.454673:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654860288:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757114.454677:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654860416:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757114.454688:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757114.454693:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757114/real 1787757114] req@ffff8c05b2d9b100 x1874588654860416/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757114.454701:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654860416/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757114.454710:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757114.454712:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757114.454714:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654860416:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757114.581964:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757115.030318:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757115.606010:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757116.054155:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757116.630000:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757117.078190:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757117.653997:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757118.102109:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757118.677993:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757119.126189:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757119.573996:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757119.574015:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787757119.574021:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757119.574034:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10175 00000100:00080000:1.0:1787757119.574047:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757119.574057:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757119.574105:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588654860544/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757119.574142:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757119.574147:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757119.574150:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757119.574155:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10175 00000100:00080000:1.0:1787757119.574159:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757119.574163:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757119.574174:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757119.574225:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588654860672/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757119.574243:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757119.574250:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757119.574254:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787757119.574254:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654860544:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757119.574269:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10175 00000100:00080000:1.0:1787757119.574279:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757119.574284:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757119.574291:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757119.574311:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588654860800/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757119.574324:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757119.574329:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757119.574331:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757119.574336:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10175 00000100:00080000:1.0:1787757119.574342:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757119.574346:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757119.574352:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757119.574365:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654860928/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757119.574378:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787757119.574384:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757119.574386:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757119.574394:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757119.574401:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10175 00000100:00080000:1.0:1787757119.574406:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757119.574411:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757119.574418:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757119.574432:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654861056/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757119.574454:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787757119.574462:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757119/real 1787757119] req@ffff8c05b2d9aa00 x1874588654860544/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757119.574463:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757119.574471:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757119.574487:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10175 00000100:00080000:1.0:1787757119.574497:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757119.574505:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757119.574512:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757119.574515:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588654860544/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757119.574532:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588654861184/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757119.574538:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757119.574555:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787757119.574559:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787757119.574567:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654860544:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757119.574569:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787757119.574577:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757119.574588:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10175 00000100:00080000:1.0:1787757119.574593:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757119.574598:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757119.574604:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757119.574615:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654860672:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757119.574622:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654861312/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757119.574654:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787757119.574662:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757119.574665:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757119.574671:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757119.574680:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10175 00000100:00080000:1.0:1787757119.574688:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787757119.574694:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757119/real 1787757119] req@ffff8c05b2d9bb80 x1874588654860672/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757119.574700:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757119.574712:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757119.574728:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588654860672/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757119.574737:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654861440/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757119.574759:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757119.574772:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757119.574779:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654860672:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757119.574790:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654860800:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757119.574820:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757119.574834:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757119/real 1787757119] req@ffff8c05b2d9b100 x1874588654860800/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757119.574849:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654860800/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757119.574920:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757119.574925:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757119.574929:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654860800:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757119.574938:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654860928:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757119.574975:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757119.574999:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757119/real 1787757119] req@ffff8c05b2d99880 x1874588654860928/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757119.575025:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654860928/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757119.575043:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757119.575049:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757119.575057:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654860928:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757119.575068:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654861056:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757119.575099:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757119.575115:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757119/real 1787757119] req@ffff8c05b2d98380 x1874588654861056/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757119.575130:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654861056/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757119.575140:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757119.575144:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757119.575148:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654861056:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757119.575154:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654861184:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757119.575173:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757119.575216:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757119/real 1787757119] req@ffff8c05b2d9b480 x1874588654861184/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757119.575242:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654861184/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757119.575258:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757119.575265:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757119.575272:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654861184:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757119.575312:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654861312:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757119.575357:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757119.575382:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757119/real 1787757119] req@ffff8c05b2d98000 x1874588654861312/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757119.575408:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654861312/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757119.575423:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757119.575427:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757119.575431:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654861312:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757119.575439:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654861440:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757119.575467:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757119.575482:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757119/real 1787757119] req@ffff8c05b2d99f80 x1874588654861440/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757119.575496:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654861440/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757119.575506:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757119.575510:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757119.575514:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654861440:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757119.701986:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757120.150107:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757120.726026:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757121.174232:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757121.750026:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757122.198134:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757122.774026:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757123.222270:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757123.798101:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757124.246025:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757124.694039:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757124.694052:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757124.694058:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757124.694083:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10181 00000100:00080000:3.0:1787757124.694148:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757124.694163:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757124.694222:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588654861568/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757124.694263:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757124.694268:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757124.694271:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757124.694277:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10181 00000100:00080000:3.0:1787757124.694283:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757124.694288:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757124.694299:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757124.694317:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588654861696/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757124.694350:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757124.694357:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757124.694361:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757124.694370:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10181 00000100:00080000:3.0:1787757124.694379:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757124.694388:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757124.694398:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654861568:192.168.201.107@tcp:250:kworker.0 00000008:00100000:3.0:1787757124.694403:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757124.694434:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588654861824/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757124.694452:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757124.694457:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757124.694461:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757124.694466:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10181 00000100:00080000:3.0:1787757124.694473:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757124.694479:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757124.694487:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787757124.694494:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757124.694506:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588654861952/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757124.694523:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757124.694528:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757124.694530:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757124.694535:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10181 00000100:00080000:3.0:1787757124.694542:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757124.694547:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757124.694552:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757124.694565:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588654862080/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757124.694577:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757124/real 1787757124] req@ffff8c05a9f79880 x1874588654861568/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757124.694583:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757124.694614:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787757124.694620:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588654861568/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757124.694621:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757124.694638:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10181 00000100:00080000:3.0:1787757124.694654:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757124.694656:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757124.694672:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:2.0:1787757124.694673:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757124.694686:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654861568:192.168.201.107@tcp:250:kworker.0 00000008:00100000:3.0:1787757124.694691:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757124.694701:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654861696:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757124.694728:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588654862208/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787757124.694741:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757124.694763:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787757124.694769:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757124/real 1787757124] req@ffff8c05a9f7a300 x1874588654861696/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757124.694771:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757124.694777:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757124.694792:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10181 00000100:00080000:3.0:1787757124.694804:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787757124.694809:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588654861696/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757124.694821:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:2.0:1787757124.694834:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787757124.694840:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787757124.694842:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757124.694855:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654861696:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757124.694944:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654861824:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757124.694944:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588654862336/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757124.694984:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787757124.694994:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757124.694996:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757124.695003:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757124.695012:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10181 00000100:00080000:3.0:1787757124.695019:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787757124.695022:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757124/real 1787757124] req@ffff8c05a9f7aa00 x1874588654861824/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757124.695030:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757124.695046:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757124.695056:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588654861824/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757124.695074:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588654862464/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757124.695124:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757124.695139:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757124.695146:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654861824:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757124.695157:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654861952:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757124.695186:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757124.695199:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757124/real 1787757124] req@ffff8c05a9f78700 x1874588654861952/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757124.695214:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588654861952/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757124.695224:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757124.695228:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757124.695232:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654861952:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757124.695238:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654862080:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757124.695261:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757124.695281:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757124/real 1787757124] req@ffff8c05a9f7b480 x1874588654862080/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757124.695305:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588654862080/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757124.695321:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757124.695338:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757124.695346:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654862080:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757124.695358:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654862208:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757124.695400:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757124.695425:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757124/real 1787757124] req@ffff8c05a9f78a80 x1874588654862208/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757124.695449:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588654862208/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757124.695465:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757124.695471:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757124.695478:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654862208:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757124.695510:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654862336:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757124.695547:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757124.695569:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757124/real 1787757124] req@ffff8c05a9f7a680 x1874588654862336/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757124.695593:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588654862336/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757124.695610:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757124.695617:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757124.695624:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654862336:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757124.695636:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654862464:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757124.695672:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757124.695694:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757124/real 1787757124] req@ffff8c05a9f78380 x1874588654862464/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757124.695718:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588654862464/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757124.695735:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757124.695741:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757124.695748:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654862464:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757124.822019:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757125.270306:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757125.846028:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757126.294151:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757126.870051:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757127.318270:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757127.893993:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757128.342179:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757128.918054:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757129.366310:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757129.814085:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757129.814113:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757129.814119:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757129.814132:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10186 00000100:00080000:3.0:1787757129.814145:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757129.814154:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757129.814240:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588654862592/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757129.814306:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757129.814311:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757129.814314:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757129.814320:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10186 00000100:00080000:3.0:1787757129.814325:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757129.814330:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757129.814341:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757129.814362:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588654862720/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757129.814375:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757129.814379:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757129.814381:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757129.814386:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10186 00000100:00080000:3.0:1787757129.814391:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757129.814394:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757129.814400:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757129.814408:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654862592:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757129.814425:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588654862848/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757129.814449:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757129.814455:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757129.814460:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757129.814469:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10186 00000100:00080000:3.0:1787757129.814478:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757129.814486:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757129.814497:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787757129.814509:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757129.814520:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588654862976/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757129.814543:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757129.814547:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757129.814551:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757129.814558:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10186 00000100:00080000:3.0:1787757129.814563:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757129.814568:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757129.814574:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757129.814609:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757129/real 1787757129] req@ffff8c05a9f78380 x1874588654862592/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757129.814672:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588654862592/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757129.814696:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757129.814707:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757129.814723:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654862592:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757129.814737:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588654863104/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757129.814753:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654862720:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757129.814757:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757129.814771:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757129.814775:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757129.814781:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10186 00000100:00080000:3.0:1787757129.814788:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757129.814794:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757129.814803:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787757129.814820:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757129.814824:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588654863232/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757129.814840:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757129.814845:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757129.814848:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757129.814864:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757129/real 1787757129] req@ffff8c05a9f7a680 x1874588654862720/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757129.814914:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10186 00000100:00080000:3.0:1787757129.814925:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757129.814931:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757129.814940:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757129.814965:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588654863360/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757129.814980:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588654862720/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757129.814991:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757129.815028:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757129.815043:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757129.815052:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757129.815058:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10186 00000100:00080000:0.0:1787757129.815069:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787757129.815072:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757129.815084:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654862720:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757129.815089:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757129.815131:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757129.815152:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588654863488/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757129.815221:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654862848:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757129.815283:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757129.815310:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757129/real 1787757129] req@ffff8c05a9f78a80 x1874588654862848/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757129.815350:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588654862848/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757129.815375:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757129.815381:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757129.815387:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654862848:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757129.815398:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654862976:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757129.815433:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757129.815452:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757129/real 1787757129] req@ffff8c05a9f7b480 x1874588654862976/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757129.815472:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588654862976/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757129.815487:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757129.815492:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757129.815498:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654862976:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757129.815509:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654863104:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757129.815541:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757129.815561:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757129/real 1787757129] req@ffff8c05a9f78700 x1874588654863104/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757129.815582:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588654863104/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757129.815597:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757129.815602:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757129.815609:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654863104:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757129.815620:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654863232:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757129.815655:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757129.815673:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757129/real 1787757129] req@ffff8c05a9f7aa00 x1874588654863232/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757129.815696:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588654863232/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757129.815735:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757129.815743:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757129.815750:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654863232:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757129.815765:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654863360:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757129.815811:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757129.815842:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757129/real 1787757129] req@ffff8c05a9f7a300 x1874588654863360/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757129.815869:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588654863360/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757129.815886:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757129.815894:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757129.815903:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654863360:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757129.815977:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654863488:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757129.816073:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757129.816096:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757129/real 1787757129] req@ffff8c05a9f79880 x1874588654863488/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757129.816120:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588654863488/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757129.816135:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757129.816141:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757129.816146:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654863488:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757129.942093:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757130.390111:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757130.966116:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757131.414299:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757131.989992:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757132.438203:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757133.014090:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757133.462306:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757134.038081:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757134.486255:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757134.934023:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757134.934040:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757134.934047:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757134.934063:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10191 00000100:00080000:3.0:1787757134.934081:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757134.934093:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757134.934146:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588654863616/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757134.934197:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757134.934205:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757134.934208:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757134.934217:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10191 00000100:00080000:3.0:1787757134.934226:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757134.934237:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757134.934254:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757134.934262:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654863616:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757134.934287:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588654863744/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757134.934305:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757134.934310:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757134.934313:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757134.934319:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10191 00000100:00080000:3.0:1787757134.934325:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757134.934330:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757134.934338:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757134.934358:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588654863872/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757134.934371:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757134.934376:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757134.934379:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757134.934384:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10191 00000100:00080000:3.0:1787757134.934391:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757134.934396:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757134.934403:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757134.934423:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654863744:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757134.934424:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588654864000/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757134.934456:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757134.934461:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757134.934464:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757134.934469:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10191 00000100:00080000:3.0:1787757134.934475:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757134.934479:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757134.934488:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757134.934493:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654863872:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757134.934511:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588654864128/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757134.934525:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757134.934531:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757134.934534:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757134.934545:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654864000:192.168.201.107@tcp:8:kworker.0 00000100:00080000:3.0:1787757134.934546:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10191 00000100:00080000:3.0:1787757134.934558:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757134.934568:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757134.934575:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757134.934597:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588654864256/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757134.934602:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654864128:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757134.934642:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757134.934647:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757134.934650:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757134.934656:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654864256:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757134.934661:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10191 00000100:00080000:3.0:1787757134.934674:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757134.934680:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757134.934687:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757134.934707:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588654864384/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757134.934724:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757134.934730:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757134.934732:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757134.934737:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10191 00000100:00080000:3.0:1787757134.934744:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757134.934749:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757134.934756:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757134.934768:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654864384:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757134.934774:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588654864512/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757134.934819:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654864512:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757135.062088:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757135.510249:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757136.086026:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757136.534018:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757137.109979:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757137.558077:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787757138.006282:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.107:988 00000400:00000100:1.0:1787757138.006338:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.107@tcp at host 192.168.201.107 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787757138.006363:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787757138.006531:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787757138.006550:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787757138.006564:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787757138.006611:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787757138.006630:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:0.0:1787757138.006642:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757134/real 1787757138] req@ffff8c05a9f7ad80 x1874588654864000/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787757138.006654:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787757138.006677:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:0.0:1787757138.006700:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588654864000/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757138.006736:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757138.006745:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757138.006752:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654864000:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757138.006765:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757134/real 1787757138] req@ffff8c05a9f7bb80 x1874588654864128/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757138.006788:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588654864128/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757138.006802:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757138.006808:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757138.006814:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654864128:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757138.006825:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757134/real 1787757138] req@ffff8c05a9f79180 x1874588654863744/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757138.006844:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588654863744/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757138.006925:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757138.006935:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757138.006941:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654863744:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757138.006954:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757134/real 1787757138] req@ffff8c05a9f79f80 x1874588654863872/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757138.006989:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588654863872/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757138.007004:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757138.007018:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757138.007024:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654863872:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757138.007035:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757134/real 1787757138] req@ffff8c05a9f78e00 x1874588654863616/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757138.007056:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588654863616/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757138.007069:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757138.007076:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757138.007082:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654863616:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787757138.007124:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757134/real 1787757138] req@ffff8c05a9f79500 x1874588654864512/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757138.007147:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588654864512/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757138.007161:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757138.007167:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757138.007173:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654864512:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757138.007183:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757134/real 1787757138] req@ffff8c05a9f7b100 x1874588654864384/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757138.007204:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588654864384/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757138.007218:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757138.007224:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757138.007230:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654864384:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757138.007241:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757134/real 1787757138] req@ffff8c05a9f7b800 x1874588654864256/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757138.007262:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588654864256/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757138.007275:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757138.007280:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757138.007286:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654864256:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757138.133998:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757138.582094:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757139.157997:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757139.606251:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757140.054026:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757140.054044:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757140.054050:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757140.054066:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10196 00000100:00080000:3.0:1787757140.054084:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757140.054097:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757140.054143:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588654864640/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757140.054185:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757140.054190:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757140.054199:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757140.054205:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10196 00000100:00080000:3.0:1787757140.054210:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757140.054215:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757140.054227:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757140.054245:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588654864768/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757140.054261:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757140.054266:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757140.054269:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757140.054273:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10196 00000100:00080000:3.0:1787757140.054277:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757140.054287:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757140.054289:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654864640:192.168.201.107@tcp:250:kworker.0 00000008:00100000:3.0:1787757140.054295:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757140.054312:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588654864896/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757140.054327:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757140.054333:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757140.054336:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757140.054340:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10196 00000100:00080000:3.0:1787757140.054348:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757140.054353:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757140.054359:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757140.054375:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588654865024/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757140.054391:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787757140.054395:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757140.054407:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757140.054413:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757140.054424:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10196 00000100:00080000:3.0:1787757140.054434:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757140.054443:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757140.054454:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757140.054478:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757140/real 1787757140] req@ffff8c05a9f7b800 x1874588654864640/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757140.054485:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588654865152/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757140.054538:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757140.054546:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787757140.054553:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588654864640/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757140.054554:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757140.054572:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10196 00000100:00080000:0.0:1787757140.054592:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757140.054598:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757140.054607:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787757140.054618:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757140.054627:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654864640:192.168.201.107@tcp:250:kworker.0 00000008:00100000:3.0:1787757140.054637:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757140.054651:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654864768:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757140.054672:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588654865280/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757140.054695:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757140.054702:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757140.054707:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757140.054718:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10196 00000800:00000100:0.0:1787757140.054721:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757140.054731:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757140.054744:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757140.054756:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757140.054767:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757140/real 1787757140] req@ffff8c05a9f7b100 x1874588654864768/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757140.054787:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588654865408/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757140.054814:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787757140.054815:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588654864768/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757140.054924:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757140.054927:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757140.054936:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10196 00000100:00080000:3.0:1787757140.054945:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757140.054952:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757140.054963:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757140.054990:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588654865536/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757140.054991:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757140.055011:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757140.055030:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654864768:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757140.055088:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654864896:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757140.055141:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757140.055169:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757140/real 1787757140] req@ffff8c05a9f79500 x1874588654864896/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757140.055196:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588654864896/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757140.055213:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757140.055220:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757140.055227:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654864896:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757140.055239:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654865024:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757140.055282:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757140.055306:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757140/real 1787757140] req@ffff8c05a9f78e00 x1874588654865024/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757140.055347:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588654865024/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757140.055364:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757140.055371:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757140.055378:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654865024:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757140.055389:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654865152:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757140.055433:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757140.055476:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757140/real 1787757140] req@ffff8c05a9f79f80 x1874588654865152/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757140.055503:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588654865152/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757140.055520:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757140.055527:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757140.055534:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654865152:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757140.055547:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654865280:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757140.055586:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757140.055608:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757140/real 1787757140] req@ffff8c05a9f79180 x1874588654865280/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757140.055631:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588654865280/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757140.055660:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757140.055667:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757140.055673:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654865280:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757140.055685:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654865408:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757140.055721:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757140.055745:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757140/real 1787757140] req@ffff8c05a9f7bb80 x1874588654865408/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757140.055768:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588654865408/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757140.055784:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757140.055790:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757140.055797:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654865408:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757140.055810:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654865536:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757140.055921:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757140.055950:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757140/real 1787757140] req@ffff8c05a9f7ad80 x1874588654865536/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757140.055987:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588654865536/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757140.056004:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757140.056010:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757140.056018:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654865536:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757140.182027:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757140.630072:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757141.205960:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757141.654153:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757142.229948:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757142.678044:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757143.253963:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757143.702220:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757144.277988:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757144.726180:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757145.174025:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757145.174038:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787757145.174043:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757145.174054:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10201 00000100:00080000:2.0:1787757145.174066:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757145.174073:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757145.174117:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588654865664/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757145.174152:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757145.174156:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757145.174158:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757145.174162:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10201 00000100:00080000:2.0:1787757145.174166:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757145.174170:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757145.174179:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757145.174193:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588654865792/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757145.174205:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757145.174209:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757145.174211:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757145.174215:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10201 00000100:00080000:2.0:1787757145.174218:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757145.174228:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757145.174235:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757145.174242:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654865664:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787757145.174257:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588654865920/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757145.174271:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757145.174275:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757145.174277:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757145.174282:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10201 00000100:00080000:2.0:1787757145.174286:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757145.174290:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757145.174295:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757145.174307:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588654866048/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757145.174317:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787757145.174320:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757145.174338:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757145.174341:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757145.174347:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10201 00000100:00080000:2.0:1787757145.174352:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757145.174356:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757145.174362:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757145.174374:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588654866176/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757145.174384:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757145/real 1787757145] req@ffff8c05b2e70000 x1874588654865664/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757145.174388:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757145.174401:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757145.174406:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757145.174420:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10201 00000100:00080000:2.0:1787757145.174425:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757145.174432:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588654865664/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757145.174432:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757145.174442:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787757145.174458:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787757145.174461:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588654866304/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757145.174468:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757145.174483:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654865664:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757145.174485:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757145.174495:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787757145.174498:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757145.174502:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654865792:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757145.174508:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10201 00000100:00080000:2.0:1787757145.174532:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757145.174548:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757145.174554:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757145.174572:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588654866432/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757145.174581:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757145.174599:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757145.174607:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787757145.174608:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757145/real 1787757145] req@ffff8c05b2e70700 x1874588654865792/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757145.174612:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757145.174626:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10201 00000100:00080000:2.0:1787757145.174638:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757145.174646:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757145.174654:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588654865792/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787757145.174656:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757145.174678:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588654866560/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757145.174703:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757145.174718:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757145.174728:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654865792:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757145.174806:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654865920:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757145.174845:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757145.174859:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757145/real 1787757145] req@ffff8c05b2e73480 x1874588654865920/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757145.174873:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588654865920/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757145.174881:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757145.174885:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757145.174888:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654865920:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757145.174936:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654866048:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757145.174958:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757145.174970:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757145/real 1787757145] req@ffff8c05b2e71500 x1874588654866048/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757145.174982:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588654866048/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757145.175001:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757145.175004:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757145.175007:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654866048:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757145.175013:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654866176:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757145.175069:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757145.175082:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757145/real 1787757145] req@ffff8c05b2e73b80 x1874588654866176/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757145.175106:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588654866176/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757145.175117:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757145.175120:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757145.175124:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654866176:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757145.175130:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654866304:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757145.175149:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757145.175161:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757145/real 1787757145] req@ffff8c05b2e71c00 x1874588654866304/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757145.175174:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588654866304/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757145.175184:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757145.175187:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757145.175191:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654866304:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757145.175198:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654866432:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757145.175225:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757145.175237:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757145/real 1787757145] req@ffff8c05b2e71880 x1874588654866432/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757145.175268:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588654866432/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757145.175283:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757145.175286:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757145.175289:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654866432:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757145.175294:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654866560:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757145.175310:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757145.175318:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757145/real 1787757145] req@ffff8c05b2e70380 x1874588654866560/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757145.175329:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588654866560/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757145.175337:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757145.175340:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757145.175343:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654866560:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757145.301991:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757145.750129:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757146.326034:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757146.774041:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757147.350039:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757147.798219:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757148.374020:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757148.822097:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757149.398093:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757149.846205:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757150.294053:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757150.294065:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757150.294070:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757150.294083:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10206 00000100:00080000:3.0:1787757150.294096:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757150.294105:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757150.294152:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588654866688/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757150.294193:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757150.294198:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757150.294201:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757150.294206:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10206 00000100:00080000:3.0:1787757150.294210:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757150.294215:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757150.294226:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757150.294240:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588654866816/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757150.294253:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757150.294257:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757150.294260:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757150.294264:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10206 00000100:00080000:3.0:1787757150.294267:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757150.294272:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757150.294279:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757150.294292:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588654866944/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757150.294320:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757150.294325:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757150.294327:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757150.294336:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10206 00000100:00100000:0.0:1787757150.294338:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654866688:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757150.294341:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757150.294354:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757150.294361:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757150.294379:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588654867072/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757150.294392:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757150.294397:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757150.294399:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757150.294405:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10206 00000100:00080000:3.0:1787757150.294411:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757150.294415:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757150.294420:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757150.294434:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588654867200/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757150.294441:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757150.294458:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757150.294463:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757150.294467:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757150.294475:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10206 00000100:00080000:3.0:1787757150.294480:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757150.294484:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757150.294490:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757150.294526:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757150/real 1787757150] req@ffff8c05a9f79880 x1874588654866688/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757150.294528:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588654867328/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757150.294601:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757150.294606:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757150.294609:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757150.294615:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588654866688/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757150.294620:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10206 00000100:00080000:3.0:1787757150.294632:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757150.294666:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757150.294674:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787757150.294678:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757150.294699:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757150.294706:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588654867456/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757150.294714:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654866688:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757150.294740:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757150.294746:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787757150.294749:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654866816:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757150.294751:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757150.294765:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10206 00000100:00080000:3.0:1787757150.294773:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757150.294777:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757150.294783:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757150.294805:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588654867584/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757150.294829:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757150.294865:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757150/real 1787757150] req@ffff8c05a9f7a300 x1874588654866816/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757150.294944:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588654866816/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757150.294956:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757150.294961:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757150.294965:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654866816:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757150.294999:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654866944:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757150.295037:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757150.295053:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757150/real 1787757150] req@ffff8c05a9f7aa00 x1874588654866944/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757150.295068:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588654866944/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757150.295078:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757150.295082:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757150.295086:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654866944:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757150.295093:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654867072:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757150.295112:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757150.295123:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757150/real 1787757150] req@ffff8c05a9f78700 x1874588654867072/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757150.295138:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588654867072/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757150.295148:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757150.295151:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757150.295155:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654867072:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757150.295161:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654867200:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757150.295180:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757150.295191:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757150/real 1787757150] req@ffff8c05a9f7b480 x1874588654867200/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757150.295205:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588654867200/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757150.295214:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757150.295217:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757150.295221:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654867200:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757150.295227:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654867328:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757150.295254:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757150.295265:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757150/real 1787757150] req@ffff8c05a9f78a80 x1874588654867328/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757150.295278:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588654867328/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757150.295296:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757150.295299:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757150.295303:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654867328:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757150.295309:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654867456:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757150.295327:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757150.295343:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757150/real 1787757150] req@ffff8c05a9f7a680 x1874588654867456/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757150.295356:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588654867456/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757150.295365:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757150.295369:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757150.295372:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654867456:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757150.295379:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654867584:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757150.295396:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757150.295406:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757150/real 1787757150] req@ffff8c05a9f78380 x1874588654867584/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757150.295419:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588654867584/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757150.295429:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757150.295432:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757150.295437:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654867584:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757150.422044:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757150.870114:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757151.446035:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757151.894251:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757152.470027:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757152.918130:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757153.494012:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757153.942283:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757154.518037:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757154.966104:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757155.414004:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757155.414020:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757155.414027:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757155.414043:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10211 00000100:00080000:3.0:1787757155.414060:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757155.414070:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757155.414119:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588654867712/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757155.414159:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757155.414163:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757155.414166:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757155.414172:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10211 00000100:00080000:3.0:1787757155.414177:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757155.414181:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757155.414192:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757155.414210:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588654867840/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757155.414225:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757155.414230:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757155.414233:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757155.414238:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10211 00000100:00080000:3.0:1787757155.414244:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757155.414252:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654867712:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757155.414254:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757155.414270:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757155.414289:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588654867968/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757155.414304:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757155.414309:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757155.414312:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757155.414316:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10211 00000100:00080000:3.0:1787757155.414322:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757155.414326:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757155.414332:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757155.414347:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588654868096/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757155.414354:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757155.414375:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757155.414382:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757155.414387:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757155.414396:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10211 00000100:00080000:3.0:1787757155.414406:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757155.414421:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757155/real 1787757155] req@ffff8c05a9f78380 x1874588654867712/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757155.414422:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757155.414442:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757155.414457:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588654867712/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757155.414473:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588654868224/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757155.414477:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757155.414492:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757155.414505:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654867712:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757155.414527:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757155.414532:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757155.414537:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757155.414548:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654867840:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757155.414576:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10211 00000100:00080000:3.0:1787757155.414589:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757155.414596:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757155.414604:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757155.414633:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588654868352/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757155.414634:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757155.414652:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757155.414657:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787757155.414664:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757155/real 1787757155] req@ffff8c05a9f7a680 x1874588654867840/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757155.414665:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757155.414682:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10211 00000100:00080000:3.0:1787757155.414694:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757155.414704:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757155.414711:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588654867840/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787757155.414714:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787757155.414735:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787757155.414739:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588654868480/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757155.414746:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757155.414759:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654867840:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757155.414777:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757155.414782:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757155.414785:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757155.414792:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10211 00000100:00080000:3.0:1787757155.414798:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757155.414803:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757155.414810:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654867968:192.168.201.107@tcp:38:kworker.0 00000008:00100000:3.0:1787757155.414814:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757155.414838:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588654868608/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757155.414858:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757155.414929:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757155/real 1787757155] req@ffff8c05a9f78a80 x1874588654867968/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757155.414947:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588654867968/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757155.414957:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757155.414962:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757155.414966:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654867968:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757155.414977:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654868096:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757155.415017:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757155.415041:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757155/real 1787757155] req@ffff8c05a9f7b480 x1874588654868096/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757155.415066:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588654868096/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757155.415084:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757155.415090:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757155.415095:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654868096:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757155.415104:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654868224:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757155.415133:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757155.415154:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757155/real 1787757155] req@ffff8c05a9f78700 x1874588654868224/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757155.415180:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588654868224/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757155.415191:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757155.415196:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757155.415200:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654868224:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757155.415206:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654868352:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757155.415228:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757155.415241:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757155/real 1787757155] req@ffff8c05a9f7aa00 x1874588654868352/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757155.415258:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588654868352/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757155.415268:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757155.415272:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757155.415276:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654868352:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757155.415281:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654868480:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757155.415299:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757155.415309:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757155/real 1787757155] req@ffff8c05a9f7a300 x1874588654868480/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757155.415323:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588654868480/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757155.415341:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757155.415344:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757155.415348:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654868480:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757155.415372:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654868608:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757155.415391:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757155.415402:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757155/real 1787757155] req@ffff8c05a9f79880 x1874588654868608/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757155.415416:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588654868608/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757155.415426:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757155.415430:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757155.415434:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654868608:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757155.542047:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757155.990200:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757156.566035:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757157.014122:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757157.589988:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757158.038180:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757158.614048:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757159.062214:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757159.638026:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757160.086138:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757160.534001:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757160.534014:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757160.534019:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757160.534031:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10216 00000100:00080000:3.0:1787757160.534045:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757160.534054:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757160.534101:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588654868736/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757160.534141:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757160.534146:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757160.534148:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757160.534153:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10216 00000100:00080000:3.0:1787757160.534158:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757160.534162:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757160.534172:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757160.534188:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588654868864/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757160.534211:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757160.534216:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757160.534219:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757160.534224:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10216 00000100:00080000:3.0:1787757160.534230:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757160.534234:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654868736:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757160.534239:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757160.534255:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757160.534270:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588654868992/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757160.534285:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757160.534290:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757160.534293:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757160.534298:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10216 00000100:00080000:3.0:1787757160.534304:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757160.534308:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757160.534313:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757160.534330:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588654869120/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757160.534337:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757160.534355:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757160.534360:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757160.534364:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757160.534372:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10216 00000100:00080000:3.0:1787757160.534377:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757160.534383:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757160.534390:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757160.534404:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588654869248/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757160.534418:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787757160.534420:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757160/real 1787757160] req@ffff8c05a9f7ad80 x1874588654868736/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757160.534427:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757160.534433:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757160.534448:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10216 00000100:00080000:3.0:1787757160.534492:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757160.534503:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588654868736/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757160.534503:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757160.534517:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787757160.534538:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787757160.534549:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588654869376/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757160.534550:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757160.534565:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654868736:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787757160.534577:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654868864:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757160.534599:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757160.534605:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757160.534608:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757160.534616:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10216 00000100:00080000:3.0:1787757160.534623:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787757160.534633:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757160.534634:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757160.534648:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757160.534664:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757160/real 1787757160] req@ffff8c05a9f7bb80 x1874588654868864/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757160.534671:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588654869504/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757160.534731:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588654868864/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757160.534742:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757160.534757:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757160.534763:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757160.534777:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10216 00000100:00080000:0.0:1787757160.534778:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757160.534786:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757160.534793:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787757160.534800:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757160.534810:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654868864:192.168.201.107@tcp:38:kworker.0 00000008:00100000:3.0:1787757160.534813:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757160.534835:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588654869632/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757160.534874:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654868992:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757160.534977:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757160.534995:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757160/real 1787757160] req@ffff8c05a9f79180 x1874588654868992/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757160.535014:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588654868992/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757160.535041:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757160.535046:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757160.535050:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654868992:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757160.535058:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654869120:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757160.535081:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757160.535095:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757160/real 1787757160] req@ffff8c05a9f79f80 x1874588654869120/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757160.535109:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588654869120/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757160.535119:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757160.535123:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757160.535127:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654869120:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757160.535133:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654869248:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757160.535151:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757160.535162:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757160/real 1787757160] req@ffff8c05a9f78e00 x1874588654869248/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757160.535176:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588654869248/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757160.535186:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757160.535189:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757160.535192:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654869248:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757160.535198:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654869376:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757160.535216:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757160.535227:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757160/real 1787757160] req@ffff8c05a9f79500 x1874588654869376/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757160.535241:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588654869376/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757160.535251:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757160.535255:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757160.535266:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654869376:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757160.535272:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654869504:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757160.535289:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757160.535300:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757160/real 1787757160] req@ffff8c05a9f7b100 x1874588654869504/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757160.535313:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588654869504/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757160.535322:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757160.535326:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757160.535330:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654869504:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757160.535336:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654869632:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757160.535352:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757160.535362:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757160/real 1787757160] req@ffff8c05a9f7b800 x1874588654869632/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757160.535376:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588654869632/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757160.535386:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757160.535389:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757160.535404:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654869632:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757160.662015:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757161.110103:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757161.686025:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757162.134183:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757162.710012:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757163.158089:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757163.734050:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757164.182162:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757164.758088:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757165.206113:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757165.654057:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757165.654073:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757165.654081:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757165.654098:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10222 00000100:00080000:3.0:1787757165.654116:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757165.654128:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757165.654210:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588654869760/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757165.654270:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757165.654277:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757165.654280:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757165.654287:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10222 00000100:00080000:3.0:1787757165.654293:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757165.654301:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757165.654317:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757165.654339:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588654869888/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757165.654395:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757165.654405:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787757165.654408:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654869760:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757165.654409:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757165.654421:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10222 00000100:00080000:3.0:1787757165.654432:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757165.654438:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757165.654445:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757165.654466:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588654870016/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757165.654480:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757165.654485:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757165.654488:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757165.654493:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10222 00000100:00080000:3.0:1787757165.654498:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757165.654502:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757165.654509:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757165.654521:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588654870144/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757165.654533:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757165.654539:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757165.654541:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757165.654546:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10222 00000100:00080000:3.0:1787757165.654551:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757165.654558:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757165.654566:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787757165.654569:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757165.654585:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588654870272/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757165.654600:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757165.654604:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757165.654607:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757165.654613:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10222 00000100:00080000:3.0:1787757165.654617:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757165.654622:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757165.654627:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757165.654642:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588654870400/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757165.654652:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757165.654656:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757165.654657:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757165.654662:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10222 00000100:00080000:3.0:1787757165.654667:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757165.654680:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757165/real 1787757165] req@ffff8c05a9f7b800 x1874588654869760/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757165.654680:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757165.654694:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757165.654716:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588654870528/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757165.654732:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757165.654737:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757165.654741:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757165.654747:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10222 00000100:00080000:3.0:1787757165.654751:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757165.654755:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757165.654760:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757165.654768:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588654869760/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757165.654779:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588654870656/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757165.654823:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757165.654834:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757165.654842:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654869760:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787757165.654931:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654869888:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757165.654980:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757165.655006:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757165/real 1787757165] req@ffff8c05a9f7b100 x1874588654869888/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757165.655029:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588654869888/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757165.655042:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757165.655047:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757165.655052:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654869888:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757165.655060:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654870016:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757165.655096:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757165.655122:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757165/real 1787757165] req@ffff8c05a9f79500 x1874588654870016/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757165.655137:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588654870016/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757165.655148:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757165.655155:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757165.655160:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654870016:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757165.655169:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654870144:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757165.655196:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757165.655225:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757165/real 1787757165] req@ffff8c05a9f78e00 x1874588654870144/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757165.655244:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588654870144/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757165.655263:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757165.655271:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757165.655278:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654870144:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757165.655293:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654870272:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757165.655323:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757165.655350:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757165/real 1787757165] req@ffff8c05a9f79f80 x1874588654870272/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757165.655398:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588654870272/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757165.655413:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757165.655419:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757165.655426:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654870272:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757165.655439:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654870400:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757165.655477:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757165.655493:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757165/real 1787757165] req@ffff8c05a9f79180 x1874588654870400/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757165.655507:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588654870400/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757165.655517:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757165.655521:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757165.655525:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654870400:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757165.655532:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654870528:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757165.655552:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757165.655563:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757165/real 1787757165] req@ffff8c05a9f7bb80 x1874588654870528/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757165.655577:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588654870528/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757165.655587:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757165.655590:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757165.655595:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654870528:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757165.655602:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654870656:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757165.655621:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757165.655632:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757165/real 1787757165] req@ffff8c05a9f7ad80 x1874588654870656/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757165.655645:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588654870656/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757165.655657:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757165.655661:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757165.655664:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654870656:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757165.782070:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757166.230225:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757166.806042:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757167.254162:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757167.830027:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757168.278239:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757168.854032:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757169.302052:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757169.878012:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757170.326199:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757170.774009:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757170.774021:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787757170.774026:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757170.774040:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10227 00000100:00080000:0.0:1787757170.774054:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757170.774067:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757170.774124:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588654870784/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757170.774174:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757170.774179:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757170.774182:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757170.774188:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10227 00000100:00080000:0.0:1787757170.774206:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757170.774214:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757170.774227:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757170.774247:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654870784:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787757170.774253:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588654870912/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757170.774274:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757170.774307:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757170.774311:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757170.774317:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10227 00000100:00080000:0.0:1787757170.774323:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757170.774329:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757170.774338:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757170.774360:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588654871040/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757170.774375:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757170.774380:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757170.774388:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787757170.774389:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757170.774404:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10227 00000100:00080000:0.0:1787757170.774411:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757170.774417:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757170.774425:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757170.774441:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588654871168/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757170.774459:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757170/real 1787757170] req@ffff8c05a9f74a80 x1874588654870784/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757170.774471:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757170.774486:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757170.774492:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757170.774508:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10227 00000100:00080000:0.0:1787757170.774520:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757170.774528:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757170.774535:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757170.774537:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588654870784/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757170.774557:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588654871296/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757170.774559:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757170.774574:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757170.774589:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654870784:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757170.774593:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757170.774603:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757170.774606:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757170.774612:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10227 00000100:00080000:0.0:1787757170.774616:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757170.774621:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757170.774630:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654870912:192.168.201.107@tcp:38:kworker.0 00000008:00100000:0.0:1787757170.774632:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757170.774668:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588654871424/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787757170.774678:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757170.774703:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787757170.774712:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757170/real 1787757170] req@ffff8c05a9f76680 x1874588654870912/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757170.774715:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787757170.774722:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757170.774740:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10227 00000100:00080000:0.0:1787757170.774752:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757170.774760:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757170.774769:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588654870912/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787757170.774773:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787757170.774801:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787757170.774804:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588654871552/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757170.774814:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757170.774828:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654870912:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757170.774838:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787757170.774847:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654871040:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757170.774849:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757170.774857:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757170.774867:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10227 00000100:00080000:0.0:1787757170.774874:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757170.774914:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757170.774920:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757170.774938:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588654871680/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787757170.774977:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757170.775010:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757170/real 1787757170] req@ffff8c05a9f77b80 x1874588654871040/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757170.775027:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588654871040/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757170.775046:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757170.775050:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757170.775062:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654871040:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757170.775071:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654871168:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757170.775096:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757170.775109:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757170/real 1787757170] req@ffff8c05a9f76d80 x1874588654871168/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757170.775123:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588654871168/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757170.775134:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757170.775138:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757170.775141:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654871168:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757170.775148:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654871296:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757170.775166:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757170.775177:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757170/real 1787757170] req@ffff8c05a9f76a00 x1874588654871296/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757170.775190:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588654871296/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757170.775200:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757170.775204:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757170.775209:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654871296:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757170.775233:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654871424:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757170.775253:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757170.775265:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757170/real 1787757170] req@ffff8c05a9f74380 x1874588654871424/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757170.775306:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588654871424/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757170.775320:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757170.775325:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757170.775329:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654871424:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757170.775337:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654871552:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757170.775366:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757170.775381:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757170/real 1787757170] req@ffff8c05a9f75c00 x1874588654871552/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757170.775395:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588654871552/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757170.775405:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757170.775410:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757170.775413:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654871552:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757170.775420:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654871680:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757170.775438:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757170.775450:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757170/real 1787757170] req@ffff8c05a9f74a80 x1874588654871680/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757170.775464:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588654871680/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757170.775473:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757170.775477:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757170.775480:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654871680:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757170.902086:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757171.350206:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757171.926090:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757172.374465:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757172.950031:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757173.398147:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757173.974025:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757174.422276:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757174.997957:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757175.446080:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757175.893992:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757175.894006:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787757175.894012:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757175.894028:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10232 00000100:00080000:0.0:1787757175.894045:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757175.894057:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757175.894116:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588654871808/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757175.894168:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757175.894173:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757175.894176:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757175.894182:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10232 00000100:00080000:0.0:1787757175.894187:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757175.894231:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757175.894243:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757175.894263:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588654871936/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757175.894277:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757175.894282:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757175.894285:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757175.894290:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10232 00000100:00080000:0.0:1787757175.894296:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757175.894300:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757175.894308:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757175.894319:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654871808:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787757175.894326:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588654872064/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757175.894347:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757175.894352:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757175.894354:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757175.894359:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10232 00000100:00080000:0.0:1787757175.894364:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757175.894369:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757175.894376:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757175.894392:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588654872192/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757175.894404:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757175.894410:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757175.894412:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757175.894417:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10232 00000100:00080000:0.0:1787757175.894423:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787757175.894427:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757175.894435:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757175.894444:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757175.894460:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588654872320/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757175.894472:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757175.894477:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757175.894479:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757175.894484:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10232 00000100:00080000:0.0:1787757175.894490:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787757175.894497:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757175/real 1787757175] req@ffff8c05a9f77800 x1874588654871808/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757175.894502:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757175.894515:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757175.894539:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588654872448/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757175.894546:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588654871808/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757175.894597:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757175.894612:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787757175.894617:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787757175.894620:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654871808:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757175.894637:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787757175.894645:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654871936:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757175.894649:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757175.894665:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10232 00000100:00080000:0.0:1787757175.894675:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787757175.894686:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757175.894689:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757175.894708:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757175/real 1787757175] req@ffff8c05a9f76300 x1874588654871936/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787757175.894717:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757175.894735:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588654871936/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757175.894750:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588654872576/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757175.894751:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757175.894767:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757175.894781:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654871936:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757175.894787:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757175.894794:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757175.894797:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757175.894813:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10232 00000100:00100000:2.0:1787757175.894820:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654872064:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757175.894822:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757175.894838:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757175.894845:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787757175.894863:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757175.894869:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588654872704/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757175.894943:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757175/real 1787757175] req@ffff8c05a9f74e00 x1874588654872064/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757175.894964:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588654872064/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757175.894977:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757175.894982:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757175.894986:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654872064:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757175.894994:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654872192:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757175.895019:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757175.895033:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757175/real 1787757175] req@ffff8c05a9f74000 x1874588654872192/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757175.895047:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588654872192/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757175.895058:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757175.895062:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757175.895066:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654872192:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757175.895072:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654872320:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757175.895090:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757175.895100:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757175/real 1787757175] req@ffff8c05a9f75880 x1874588654872320/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757175.895114:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588654872320/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757175.895124:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757175.895128:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757175.895131:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654872320:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757175.895137:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654872448:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757175.895155:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757175.895173:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757175/real 1787757175] req@ffff8c05a9f75500 x1874588654872448/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757175.895220:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588654872448/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757175.895237:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757175.895244:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757175.895252:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654872448:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757175.895265:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654872576:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757175.895305:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757175.895323:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757175/real 1787757175] req@ffff8c05a9f77100 x1874588654872576/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757175.895339:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588654872576/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757175.895349:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757175.895353:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757175.895357:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654872576:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757175.895381:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654872704:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757175.895403:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757175.895415:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757175/real 1787757175] req@ffff8c05a9f75180 x1874588654872704/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757175.895429:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588654872704/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757175.895439:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757175.895442:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757175.895446:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654872704:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757176.022064:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757176.470164:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757177.046030:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757177.494092:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757178.070087:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757178.518287:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757179.094084:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757179.542159:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757180.117999:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757180.566257:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757181.013988:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757181.014001:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787757181.014007:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757181.014020:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10237 00000100:00080000:0.0:1787757181.014035:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757181.014044:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757181.014095:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588654872832/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757181.014177:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757181.014182:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757181.014185:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757181.014191:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10237 00000100:00080000:0.0:1787757181.014196:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757181.014201:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757181.014214:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757181.014235:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588654872960/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757181.014251:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757181.014255:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757181.014262:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787757181.014265:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654872832:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757181.014271:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10237 00000100:00080000:0.0:1787757181.014284:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757181.014288:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757181.014295:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757181.014313:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588654873088/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757181.014326:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757181.014332:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757181.014335:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757181.014340:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10237 00000100:00080000:0.0:1787757181.014351:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757181.014361:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:2.0:1787757181.014365:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:0.0:1787757181.014383:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757181.014412:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588654873216/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757181.014429:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757181/real 1787757181] req@ffff8c05a9f75180 x1874588654872832/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757181.014445:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757181.014456:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757181.014460:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787757181.014467:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588654872832/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757181.014474:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10237 00000100:00080000:0.0:1787757181.014488:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757181.014496:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757181.014498:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:2.0:1787757181.014511:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000008:00100000:0.0:1787757181.014514:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757181.014520:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654872832:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787757181.014542:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588654873344/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757181.014561:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654872960:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757181.014562:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757181.014579:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757181.014581:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757181.014587:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10237 00000100:00080000:0.0:1787757181.014591:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757181.014596:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757181.014604:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757181.014625:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588654873472/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787757181.014640:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757181.014655:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757181.014660:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787757181.014663:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787757181.014670:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757181/real 1787757181] req@ffff8c05a9f77100 x1874588654872960/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757181.014672:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10237 00000100:00080000:0.0:1787757181.014683:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757181.014696:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757181.014707:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757181.014723:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588654872960/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757181.014726:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588654873600/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757181.014763:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757181.014770:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757181.014776:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787757181.014781:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757181.014791:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787757181.014792:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654872960:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757181.014807:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10237 00000100:00100000:2.0:1787757181.014814:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654873088:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757181.014817:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757181.014833:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757181.014841:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757181.014861:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588654873728/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787757181.014943:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757181.014979:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757181/real 1787757181] req@ffff8c05a9f75500 x1874588654873088/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757181.015008:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588654873088/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757181.015027:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757181.015032:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757181.015036:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654873088:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757181.015046:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654873216:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757181.015094:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757181.015143:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757181/real 1787757181] req@ffff8c05a9f75880 x1874588654873216/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757181.015160:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588654873216/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757181.015171:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757181.015175:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757181.015179:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654873216:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757181.015202:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654873344:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757181.015229:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757181.015243:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757181/real 1787757181] req@ffff8c05a9f74000 x1874588654873344/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757181.015266:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588654873344/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757181.015276:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757181.015279:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757181.015283:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654873344:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757181.015290:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654873472:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757181.015309:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757181.015320:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757181/real 1787757181] req@ffff8c05a9f74e00 x1874588654873472/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757181.015334:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588654873472/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757181.015344:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757181.015347:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757181.015351:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654873472:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757181.015357:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654873600:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757181.015374:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757181.015384:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757181/real 1787757181] req@ffff8c05a9f76300 x1874588654873600/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757181.015398:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588654873600/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757181.015415:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757181.015419:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757181.015423:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654873600:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757181.015429:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654873728:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757181.015447:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757181.015457:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757181/real 1787757181] req@ffff8c05a9f77800 x1874588654873728/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757181.015470:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588654873728/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757181.015480:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757181.015484:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757181.015488:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654873728:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757181.142020:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757181.590110:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757182.165996:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757182.614239:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757183.190043:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757183.638217:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757184.214025:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757184.662238:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757185.238001:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757185.686174:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757186.134074:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757186.134091:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787757186.134098:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757186.134116:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10242 00000100:00080000:0.0:1787757186.134135:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757186.134146:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757186.134211:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588654873856/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757186.134255:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757186.134260:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757186.134263:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757186.134269:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10242 00000100:00080000:0.0:1787757186.134274:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757186.134279:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757186.134294:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757186.134313:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588654873984/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757186.134327:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757186.134332:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757186.134335:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757186.134341:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10242 00000100:00080000:0.0:1787757186.134345:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757186.134352:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757186.134361:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654873856:192.168.201.107@tcp:250:kworker.0 00000008:00100000:0.0:1787757186.134362:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757186.134395:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588654874112/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757186.134417:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757186.134422:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757186.134425:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757186.134433:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10242 00000100:00080000:0.0:1787757186.134438:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757186.134446:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757186.134457:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787757186.134476:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757186.134488:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588654874240/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757186.134515:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757186.134524:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757186.134530:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787757186.134537:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757186/real 1787757186] req@ffff8c05a9f74a80 x1874588654873856/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757186.134546:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10242 00000100:00080000:0.0:1787757186.134569:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757186.134580:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757186.134590:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588654873856/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787757186.134591:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787757186.134629:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787757186.134630:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588654874368/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757186.134645:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757186.134661:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654873856:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757186.134691:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787757186.134693:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654873984:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757186.134727:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757186.134735:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757186.134747:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10242 00000100:00080000:0.0:1787757186.134755:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757186.134767:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:2.0:1787757186.134770:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:0.0:1787757186.134778:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757186.134803:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588654874496/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757186.134812:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757186/real 1787757186] req@ffff8c05a9f75c00 x1874588654873984/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757186.134837:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757186.134842:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787757186.134844:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757186.134865:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10242 00000100:00080000:0.0:1787757186.134943:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757186.134954:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757186.134963:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757186.134964:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588654873984/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757186.134987:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787757186.134990:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588654874624/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757186.135036:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787757186.135037:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757186.135047:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787757186.135056:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654873984:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757186.135057:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757186.135092:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10242 00000100:00080000:0.0:1787757186.135100:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757186.135107:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757186.135117:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757186.135130:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654874112:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757186.135142:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588654874752/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787757186.135181:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757186.135207:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757186/real 1787757186] req@ffff8c05a9f74380 x1874588654874112/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757186.135224:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588654874112/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757186.135235:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757186.135240:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757186.135244:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654874112:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757186.135253:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654874240:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757186.135275:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757186.135287:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757186/real 1787757186] req@ffff8c05a9f76a00 x1874588654874240/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757186.135302:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588654874240/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757186.135312:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757186.135315:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757186.135319:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654874240:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757186.135326:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654874368:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757186.135346:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757186.135357:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757186/real 1787757186] req@ffff8c05a9f76d80 x1874588654874368/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757186.135371:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588654874368/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757186.135381:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757186.135385:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757186.135388:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654874368:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757186.135395:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654874496:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757186.135414:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757186.135425:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757186/real 1787757186] req@ffff8c05a9f77b80 x1874588654874496/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757186.135440:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588654874496/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757186.135450:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757186.135453:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757186.135457:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654874496:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757186.135463:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654874624:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757186.135482:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757186.135493:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757186/real 1787757186] req@ffff8c05a9f76680 x1874588654874624/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757186.135507:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588654874624/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757186.135517:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757186.135521:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757186.135524:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654874624:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757186.135546:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654874752:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757186.135565:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757186.135583:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757186/real 1787757186] req@ffff8c05a9f75c00 x1874588654874752/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757186.135608:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588654874752/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757186.135639:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757186.135646:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757186.135652:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654874752:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757186.261995:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757186.710413:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757187.286072:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757187.734141:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757188.310015:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757188.758148:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757189.334025:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757189.782106:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757190.358055:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757190.806180:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757191.254059:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757191.254072:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787757191.254077:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757191.254090:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10247 00000100:00080000:0.0:1787757191.254103:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757191.254112:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757191.254165:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588654874880/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757191.254207:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757191.254211:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757191.254212:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757191.254216:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10247 00000100:00080000:0.0:1787757191.254219:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757191.254223:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757191.254234:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757191.254257:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588654875008/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757191.254267:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757191.254270:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757191.254272:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757191.254275:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10247 00000100:00080000:0.0:1787757191.254278:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757191.254285:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757191.254287:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654874880:192.168.201.107@tcp:250:kworker.0 00000008:00100000:0.0:1787757191.254290:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757191.254322:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588654875136/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757191.254331:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757191.254335:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757191.254337:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757191.254340:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10247 00000100:00080000:0.0:1787757191.254343:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757191.254346:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757191.254350:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757191.254360:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588654875264/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757191.254368:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757191.254372:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757191.254373:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757191.254376:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10247 00000100:00080000:0.0:1787757191.254379:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757191.254382:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757191.254385:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787757191.254392:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757191.254397:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588654875392/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757191.254424:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757191.254430:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757191.254432:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757191.254439:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10247 00000100:00080000:0.0:1787757191.254448:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757191.254453:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757191.254459:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757191.254474:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588654875520/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757191.254485:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787757191.254493:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757191/real 1787757191] req@ffff8c05a9f74a80 x1874588654874880/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757191.254494:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787757191.254500:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757191.254510:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10247 00000100:00080000:0.0:1787757191.254516:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757191.254527:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757191.254539:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757191.254579:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588654874880/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757191.254580:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588654875648/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757191.254614:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757191.254616:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757191.254622:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787757191.254644:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787757191.254651:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654874880:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757191.254652:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757191.254669:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10247 00000100:00100000:2.0:1787757191.254677:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654875008:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757191.254679:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757191.254694:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757191.254704:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787757191.254734:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757191.254741:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588654875776/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757191.254767:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757191/real 1787757191] req@ffff8c05a9f77800 x1874588654875008/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757191.254806:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588654875008/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757191.254819:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757191.254824:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757191.254829:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654875008:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757191.254857:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654875136:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757191.254876:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757191.254885:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757191/real 1787757191] req@ffff8c05a9f76300 x1874588654875136/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757191.254894:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588654875136/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757191.254901:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757191.254903:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757191.254906:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654875136:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757191.254965:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654875264:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757191.254986:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757191.254995:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757191/real 1787757191] req@ffff8c05a9f74e00 x1874588654875264/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757191.255004:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588654875264/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757191.255011:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757191.255013:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757191.255016:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654875264:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757191.255021:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654875392:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757191.255034:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757191.255044:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757191/real 1787757191] req@ffff8c05a9f74000 x1874588654875392/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757191.255058:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588654875392/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757191.255067:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757191.255071:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757191.255076:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654875392:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757191.255082:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654875520:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757191.255106:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757191.255116:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757191/real 1787757191] req@ffff8c05a9f75880 x1874588654875520/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757191.255126:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588654875520/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757191.255133:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757191.255135:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757191.255137:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654875520:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757191.255142:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654875648:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757191.255156:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757191.255164:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757191/real 1787757191] req@ffff8c05a9f75500 x1874588654875648/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757191.255173:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588654875648/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757191.255189:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757191.255191:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757191.255193:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654875648:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757191.255197:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654875776:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757191.255209:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757191.255215:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757191/real 1787757191] req@ffff8c05a9f77100 x1874588654875776/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757191.255224:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588654875776/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757191.255230:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757191.255232:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757191.255234:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654875776:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757191.382035:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757191.830182:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757192.405986:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757192.854189:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757193.431049:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757193.878120:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757194.454035:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757194.902201:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757195.478039:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757195.926140:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757196.373978:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757196.373991:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787757196.373997:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757196.374009:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10252 00000100:00080000:0.0:1787757196.374022:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757196.374031:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757196.374096:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588654875904/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757196.374131:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757196.374136:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757196.374138:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757196.374144:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10252 00000100:00080000:0.0:1787757196.374148:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757196.374153:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757196.374164:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757196.374181:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588654876032/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757196.374196:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757196.374201:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757196.374205:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757196.374210:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10252 00000100:00080000:0.0:1787757196.374214:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757196.374219:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757196.374225:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757196.374244:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588654876160/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757196.374248:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654875904:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757196.374266:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757196.374270:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757196.374272:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757196.374277:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10252 00000100:00080000:0.0:1787757196.374288:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757196.374294:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757196.374299:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757196.374315:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588654876288/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757196.374327:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757196.374331:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757196.374334:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757196.374338:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10252 00000100:00080000:0.0:1787757196.374343:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757196.374348:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757196.374354:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757196.374371:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588654876416/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787757196.374381:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757196.374398:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757196.374402:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757196.374406:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757196.374412:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10252 00000100:00080000:0.0:1787757196.374417:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757196.374421:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757196.374427:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757196.374444:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588654876544/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757196.374456:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787757196.374464:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757196/real 1787757196] req@ffff8c05a9f75180 x1874588654875904/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757196.374465:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787757196.374472:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757196.374488:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10252 00000100:00080000:0.0:1787757196.374499:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757196.374507:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757196.374513:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757196.374521:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588654875904/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757196.374531:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588654876672/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757196.374555:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757196.374558:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757196.374566:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787757196.374568:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757196.374576:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787757196.374580:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654875904:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757196.374588:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10252 00000100:00080000:0.0:1787757196.374595:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757196.374599:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757196.374605:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757196.374608:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654876032:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757196.374623:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588654876800/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787757196.374652:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757196.374671:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757196/real 1787757196] req@ffff8c05a9f77100 x1874588654876032/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757196.374692:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588654876032/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757196.374704:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757196.374709:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757196.374713:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654876032:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757196.374743:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654876160:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757196.374766:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757196.374778:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757196/real 1787757196] req@ffff8c05a9f75500 x1874588654876160/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757196.374793:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588654876160/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757196.374811:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757196.374815:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757196.374860:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654876160:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757196.374869:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654876288:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757196.374939:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757196.374956:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757196/real 1787757196] req@ffff8c05a9f75880 x1874588654876288/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757196.374972:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588654876288/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757196.374982:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757196.374986:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757196.374990:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654876288:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757196.374998:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654876416:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757196.375019:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757196.375031:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757196/real 1787757196] req@ffff8c05a9f74000 x1874588654876416/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757196.375045:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588654876416/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757196.375054:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757196.375058:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757196.375063:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654876416:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757196.375069:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654876544:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757196.375088:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757196.375099:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757196/real 1787757196] req@ffff8c05a9f74e00 x1874588654876544/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757196.375112:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588654876544/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757196.375122:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757196.375126:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757196.375131:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654876544:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757196.375142:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654876672:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757196.375178:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757196.375200:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757196/real 1787757196] req@ffff8c05a9f76300 x1874588654876672/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757196.375226:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588654876672/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757196.375243:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757196.375249:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757196.375255:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654876672:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757196.375263:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654876800:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757196.375292:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757196.375307:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757196/real 1787757196] req@ffff8c05a9f77800 x1874588654876800/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757196.375322:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588654876800/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757196.375332:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757196.375335:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757196.375339:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654876800:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757196.502064:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757196.950225:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757197.526000:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757197.974181:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757198.550060:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757198.998293:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757199.574064:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757200.022192:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757200.597959:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757201.046217:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757201.493952:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757201.493964:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787757201.493970:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757201.493984:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10257 00000100:00080000:0.0:1787757201.493998:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757201.494010:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757201.494071:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588654876928/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757201.494115:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757201.494121:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757201.494125:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757201.494134:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10257 00000100:00080000:0.0:1787757201.494140:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757201.494154:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757201.494167:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757201.494192:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588654877056/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757201.494209:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654876928:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757201.494214:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757201.494228:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757201.494233:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757201.494241:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10257 00000100:00080000:0.0:1787757201.494247:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757201.494253:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757201.494261:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757201.494285:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588654877184/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757201.494300:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757201.494305:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757201.494308:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757201.494313:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10257 00000100:00080000:0.0:1787757201.494318:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757201.494324:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757201.494331:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757201.494344:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588654877312/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757201.494357:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757201.494361:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757201.494364:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757201.494368:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10257 00000100:00080000:0.0:1787757201.494374:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757201.494379:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757201.494386:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757201.494399:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588654877440/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757201.494412:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757201.494417:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757201.494419:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757201.494424:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10257 00000100:00080000:0.0:1787757201.494429:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757201.494436:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757201.494442:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757201.494460:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588654877568/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757201.494480:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757201.494485:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787757201.494490:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757201.494500:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10257 00000100:00080000:0.0:1787757201.494510:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757201.494521:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757201.494534:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757201.494563:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588654877696/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757201.494580:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654877056:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757201.494615:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757201.494621:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757201.494625:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757201.494632:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10257 00000100:00080000:0.0:1787757201.494639:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757201.494645:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757201.494669:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757201.494684:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654877184:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757201.494697:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588654877824/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757201.494736:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654877312:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757201.494800:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654877440:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757201.494822:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654877568:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757201.494852:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654877696:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757201.494953:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654877824:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757201.622039:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757202.070145:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757202.645985:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757203.094235:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757203.670014:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757204.118177:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787757204.566415:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.107:988 00000400:00000100:1.0:1787757204.566492:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.107@tcp at host 192.168.201.107 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787757204.566540:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787757204.567040:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:2.0:1787757204.567076:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757201/real 1787757204] req@ffff8c05a9f74a80 x1874588654876928/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787757204.567103:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787757204.567130:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787757204.567167:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:2.0:1787757204.567200:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588654876928/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787757204.567211:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787757204.567258:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00080000:2.0:1787757204.567281:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757204.567305:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000800:00000100:1.0:1787757204.567309:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:2.0:1787757204.567323:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654876928:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787757204.567378:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757201/real 1787757204] req@ffff8c05a9f74380 x1874588654877696/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757204.567414:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588654877696/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757204.567432:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757204.567440:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757204.567448:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654877696:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757204.567464:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757201/real 1787757204] req@ffff8c05a9f77800 x1874588654877824/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757204.567494:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588654877824/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757204.567512:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757204.567520:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757204.567527:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654877824:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757204.567542:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757201/real 1787757204] req@ffff8c05a9f75c00 x1874588654877056/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757204.567569:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588654877056/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757204.567585:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757204.567592:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757204.567598:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654877056:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757204.567611:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757201/real 1787757204] req@ffff8c05a9f76680 x1874588654877184/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757204.567634:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588654877184/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757204.567648:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757204.567655:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757204.567660:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654877184:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757204.567673:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757201/real 1787757204] req@ffff8c05a9f77b80 x1874588654877312/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757204.567728:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588654877312/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757204.567743:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757204.567750:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757204.567783:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654877312:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757204.567798:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757201/real 1787757204] req@ffff8c05a9f76d80 x1874588654877440/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757204.567823:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588654877440/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757204.567839:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757204.567846:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757204.567854:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654877440:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757204.567868:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757201/real 1787757204] req@ffff8c05a9f76a00 x1874588654877568/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757204.568013:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588654877568/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757204.568031:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757204.568050:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757204.568058:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654877568:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757204.693981:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757205.142308:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757205.718013:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757206.166108:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757206.614001:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757206.614014:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787757206.614020:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757206.614033:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10263 00000100:00080000:0.0:1787757206.614046:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757206.614056:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757206.614105:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588654877952/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757206.614147:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757206.614154:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757206.614158:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757206.614165:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10263 00000100:00080000:0.0:1787757206.614173:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757206.614181:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757206.614198:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757206.614225:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588654878080/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757206.614246:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787757206.614250:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654877952:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757206.614251:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757206.614255:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757206.614262:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10263 00000100:00080000:0.0:1787757206.614268:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757206.614274:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757206.614283:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757206.614300:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588654878208/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757206.614316:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757206.614322:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757206.614325:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757206.614332:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10263 00000100:00080000:0.0:1787757206.614337:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787757206.614342:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757206.614344:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757206.614367:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757206.614387:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588654878336/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757206.614403:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757206/real 1787757206] req@ffff8c05a9f76300 x1874588654877952/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757206.614404:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757206.614411:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757206.614416:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757206.614422:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10263 00000100:00080000:0.0:1787757206.614428:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787757206.614435:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588654877952/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757206.614436:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757206.614444:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787757206.614456:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787757206.614461:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588654878464/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757206.614463:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757206.614471:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654877952:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757206.614480:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757206.614485:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757206.614488:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757206.614494:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10263 00000100:00080000:0.0:1787757206.614499:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787757206.614505:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654878080:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757206.614506:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757206.614514:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757206.614531:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588654878592/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787757206.614545:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757206.614551:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757206.614557:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787757206.614561:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757206.614567:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10263 00000100:00100000:2.0:1787757206.614568:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757206/real 1787757206] req@ffff8c05a9f74e00 x1874588654878080/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757206.614573:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757206.614580:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757206.614589:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757206.614592:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588654878080/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757206.614608:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588654878720/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757206.614623:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757206.614630:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787757206.614635:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787757206.614637:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654878080:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757206.614642:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757206.614645:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787757206.614683:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654878208:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757206.614684:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10263 00000100:00080000:0.0:1787757206.614691:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757206.614699:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757206.614708:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787757206.614723:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757206.614728:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588654878848/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757206.614746:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757206/real 1787757206] req@ffff8c05a9f74000 x1874588654878208/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757206.614810:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588654878208/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757206.614834:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757206.614841:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757206.614848:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654878208:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757206.614861:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654878336:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757206.614972:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757206.614998:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757206/real 1787757206] req@ffff8c05a9f75880 x1874588654878336/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757206.615025:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588654878336/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757206.615042:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757206.615048:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757206.615055:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654878336:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757206.615067:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654878464:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757206.615110:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757206.615134:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757206/real 1787757206] req@ffff8c05a9f75500 x1874588654878464/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757206.615159:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588654878464/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757206.615190:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757206.615197:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757206.615204:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654878464:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757206.615241:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654878592:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757206.615285:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757206.615308:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757206/real 1787757206] req@ffff8c05a9f77100 x1874588654878592/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757206.615331:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588654878592/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757206.615348:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757206.615353:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757206.615359:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654878592:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757206.615369:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654878720:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757206.615408:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757206.615430:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757206/real 1787757206] req@ffff8c05a9f75180 x1874588654878720/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757206.615454:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588654878720/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757206.615470:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757206.615476:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757206.615484:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654878720:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757206.615497:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654878848:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757206.615537:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757206.615559:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757206/real 1787757206] req@ffff8c05a9f76300 x1874588654878848/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757206.615584:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588654878848/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757206.615601:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757206.615608:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757206.615615:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654878848:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757206.741994:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757207.190315:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757207.766030:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757208.214103:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757208.790019:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757209.238176:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757209.814015:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757210.262079:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757210.837994:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757211.286202:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757211.734052:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757211.734069:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787757211.734077:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757211.734092:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10268 00000100:00080000:1.0:1787757211.734108:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757211.734120:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757211.734177:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588654878976/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757211.734225:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757211.734233:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757211.734237:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757211.734246:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10268 00000100:00080000:1.0:1787757211.734254:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757211.734263:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757211.734279:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757211.734309:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588654879104/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757211.734332:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757211.734339:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757211.734342:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757211.734349:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10268 00000100:00080000:1.0:1787757211.734355:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787757211.734365:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654878976:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757211.734368:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757211.734385:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757211.734407:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654879232/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757211.734423:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757211.734428:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757211.734431:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757211.734436:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10268 00000100:00080000:1.0:1787757211.734441:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757211.734446:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757211.734453:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787757211.734470:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757211.734472:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654879360/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757211.734490:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757211.734495:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757211.734499:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757211.734505:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10268 00000100:00080000:1.0:1787757211.734509:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757211.734514:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757211.734520:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757211.734538:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588654879488/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757211.734546:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757211/real 1787757211] req@ffff8c05b2d9a300 x1874588654878976/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757211.734588:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757211.734601:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757211.734605:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757211.734619:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10268 00000100:00080000:1.0:1787757211.734629:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757211.734637:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757211.734645:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757211.734657:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588654878976/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757211.734665:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654879616/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757211.734720:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757211.734724:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757211.734725:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787757211.734732:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757211.734739:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787757211.734742:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10268 00000100:00080000:1.0:1787757211.734751:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787757211.734755:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654878976:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757211.734763:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757211.734774:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757211.734781:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654879104:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757211.734792:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654879744/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757211.734806:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757211.734810:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757211.734813:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757211.734819:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10268 00000100:00080000:1.0:1787757211.734826:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757211.734835:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:2.0:1787757211.734837:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:1.0:1787757211.734843:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757211.734863:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588654879872/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757211.734951:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757211/real 1787757211] req@ffff8c05b2d98700 x1874588654879104/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757211.734989:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654879104/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757211.735001:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757211.735006:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757211.735010:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654879104:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757211.735043:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654879232:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757211.735076:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757211.735093:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757211/real 1787757211] req@ffff8c05b2d99f80 x1874588654879232/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757211.735108:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654879232/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757211.735118:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757211.735122:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757211.735126:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654879232:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757211.735133:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654879360:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757211.735153:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757211.735164:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757211/real 1787757211] req@ffff8c05b2d98000 x1874588654879360/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757211.735178:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654879360/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757211.735188:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757211.735192:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757211.735196:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654879360:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757211.735202:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654879488:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757211.735221:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757211.735232:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757211/real 1787757211] req@ffff8c05b2d9b480 x1874588654879488/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757211.735245:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654879488/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757211.735255:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757211.735259:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757211.735262:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654879488:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757211.735268:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654879616:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757211.735287:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757211.735297:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757211/real 1787757211] req@ffff8c05b2d98380 x1874588654879616/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757211.735310:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654879616/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757211.735319:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757211.735323:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757211.735328:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654879616:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757211.735334:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654879744:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757211.735353:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757211.735364:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757211/real 1787757211] req@ffff8c05b2d99880 x1874588654879744/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757211.735377:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654879744/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757211.735387:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757211.735390:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757211.735394:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654879744:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757211.735400:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654879872:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757211.735419:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757211.735430:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757211/real 1787757211] req@ffff8c05b2d9b100 x1874588654879872/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757211.735444:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654879872/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757211.735453:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757211.735457:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757211.735460:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654879872:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757211.861993:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757212.310098:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757212.886041:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757213.334228:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757213.910038:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757214.358165:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757214.934057:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757215.382255:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757215.958020:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757216.406211:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757216.854081:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757216.854099:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787757216.854106:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757216.854122:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10273 00000100:00080000:2.0:1787757216.854141:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757216.854153:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757216.854219:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73800 x1874588654880000/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757216.854275:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757216.854280:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757216.854284:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757216.854292:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10273 00000100:00080000:2.0:1787757216.854299:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757216.854308:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757216.854326:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757216.854360:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588654880128/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757216.854368:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654880000:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757216.854403:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757216.854409:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757216.854412:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757216.854420:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10273 00000100:00080000:2.0:1787757216.854426:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757216.854432:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757216.854441:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757216.854460:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588654880256/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757216.854511:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757216.854516:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757216.854518:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757216.854524:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10273 00000100:00080000:2.0:1787757216.854530:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787757216.854535:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757216.854542:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757216.854553:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757216.854571:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588654880384/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757216.854585:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757216.854590:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757216.854592:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757216.854597:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10273 00000100:00080000:2.0:1787757216.854602:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757216.854608:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757216.854614:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757216.854622:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757216/real 1787757216] req@ffff8c05b2e73800 x1874588654880000/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757216.854633:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588654880512/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757216.854665:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757216.854670:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757216.854673:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757216.854681:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73800 x1874588654880000/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757216.854684:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10273 00000100:00080000:2.0:1787757216.854695:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757216.854709:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757216.854720:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787757216.854722:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757216.854733:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757216.854738:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588654880640/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757216.854748:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654880000:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757216.854762:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757216.854783:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787757216.854786:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757216.854793:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10273 00000100:00080000:2.0:1787757216.854798:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787757216.854803:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654880128:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757216.854810:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757216.854821:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757216.854840:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588654880768/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757216.854855:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757216.854864:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:3.0:1787757216.854866:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757216.854868:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757216.854928:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10273 00000100:00080000:2.0:1787757216.854934:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757216.854939:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757216.854946:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757216.854965:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588654880896/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757216.854997:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757216/real 1787757216] req@ffff8c05b2e70380 x1874588654880128/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757216.855031:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588654880128/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757216.855044:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757216.855048:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757216.855053:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654880128:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757216.855063:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654880256:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757216.855097:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757216.855112:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757216/real 1787757216] req@ffff8c05b2e71880 x1874588654880256/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757216.855127:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588654880256/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757216.855138:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757216.855143:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757216.855147:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654880256:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757216.855155:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654880384:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757216.855182:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757216.855203:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757216/real 1787757216] req@ffff8c05b2e71c00 x1874588654880384/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757216.855228:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588654880384/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757216.855259:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757216.855266:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757216.855273:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654880384:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757216.855286:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654880512:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757216.855318:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757216.855334:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757216/real 1787757216] req@ffff8c05b2e73b80 x1874588654880512/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757216.855349:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588654880512/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757216.855361:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757216.855365:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757216.855369:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654880512:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757216.855377:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654880640:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757216.855402:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757216.855422:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757216/real 1787757216] req@ffff8c05b2e71500 x1874588654880640/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757216.855447:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588654880640/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757216.855465:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757216.855498:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757216.855507:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654880640:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757216.855544:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654880768:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757216.855578:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757216.855593:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757216/real 1787757216] req@ffff8c05b2e73480 x1874588654880768/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757216.855609:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588654880768/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757216.855620:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757216.855624:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757216.855629:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654880768:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757216.855636:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654880896:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757216.855656:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757216.855667:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757216/real 1787757216] req@ffff8c05b2e70700 x1874588654880896/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757216.855682:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588654880896/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757216.855692:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757216.855696:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757216.855699:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654880896:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757216.982072:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757217.430288:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757218.006069:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757218.454137:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757219.030010:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757219.478259:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757220.054066:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757220.502172:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757221.078040:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757221.526143:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757221.974046:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757221.974058:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787757221.974063:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757221.974074:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10278 00000100:00080000:2.0:1787757221.974087:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757221.974095:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757221.974140:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588654881024/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757221.974175:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757221.974179:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757221.974182:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757221.974186:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10278 00000100:00080000:2.0:1787757221.974190:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757221.974193:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757221.974203:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757221.974219:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588654881152/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757221.974232:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757221.974236:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757221.974238:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757221.974242:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10278 00000100:00080000:2.0:1787757221.974245:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757221.974250:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757221.974257:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757221.974268:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654881024:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787757221.974283:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588654881280/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757221.974304:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757221.974308:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757221.974311:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757221.974316:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10278 00000100:00080000:2.0:1787757221.974319:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757221.974323:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757221.974328:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757221.974342:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588654881408/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757221.974353:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757221.974357:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757221.974358:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757221.974361:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10278 00000100:00080000:2.0:1787757221.974366:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787757221.974373:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757221.974400:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757221.974407:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757221.974422:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588654881536/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757221.974432:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757221.974436:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757221.974439:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757221.974445:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10278 00000100:00080000:2.0:1787757221.974450:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757221.974459:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757221.974460:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757221/real 1787757221] req@ffff8c05b2e70000 x1874588654881024/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787757221.974470:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757221.974491:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588654881664/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757221.974500:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588654881024/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757221.974524:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757221.974531:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757221.974534:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757221.974535:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757221.974548:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787757221.974550:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10278 00000100:00080000:2.0:1787757221.974558:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787757221.974560:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654881024:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757221.974568:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757221.974575:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757221.974579:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654881152:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757221.974591:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588654881792/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757221.974604:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757221.974608:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757221.974611:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787757221.974620:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757221.974621:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10278 00000100:00080000:2.0:1787757221.974632:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757221.974638:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757221.974645:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757221.974649:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757221/real 1787757221] req@ffff8c05b2e73100 x1874588654881152/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757221.974664:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588654881920/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757221.974672:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588654881152/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757221.974704:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757221.974709:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757221.974713:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654881152:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757221.974755:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654881280:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757221.974792:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757221.974814:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757221/real 1787757221] req@ffff8c05b2e72d80 x1874588654881280/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757221.974849:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588654881280/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757221.974865:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757221.974871:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757221.974927:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654881280:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757221.974939:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654881408:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757221.974980:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757221.975003:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757221/real 1787757221] req@ffff8c05b2e72680 x1874588654881408/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757221.975025:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588654881408/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757221.975056:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757221.975063:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757221.975069:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654881408:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757221.975081:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654881536:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757221.975138:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757221.975161:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757221/real 1787757221] req@ffff8c05b2e70e00 x1874588654881536/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757221.975201:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588654881536/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757221.975229:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757221.975253:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757221.975258:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654881536:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757221.975270:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654881664:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757221.975321:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757221.975337:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757221/real 1787757221] req@ffff8c05b2e72a00 x1874588654881664/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757221.975351:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588654881664/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757221.975361:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757221.975365:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757221.975413:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654881664:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757221.975422:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654881792:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757221.975457:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757221.975471:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757221/real 1787757221] req@ffff8c05b2e71180 x1874588654881792/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757221.975486:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588654881792/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757221.975496:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757221.975499:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757221.975504:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654881792:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757221.975511:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654881920:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757221.975530:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757221.975541:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757221/real 1787757221] req@ffff8c05b2e72300 x1874588654881920/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757221.975554:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588654881920/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757221.975564:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757221.975567:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757221.975571:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654881920:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757222.101958:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757222.550090:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757223.126035:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757223.574249:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757224.149985:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757224.598244:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757225.174030:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757225.622221:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757226.197991:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757226.646111:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757227.094101:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757227.094118:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787757227.094126:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757227.094143:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10283 00000100:00080000:2.0:1787757227.094161:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757227.094172:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757227.094254:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588654882048/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757227.094339:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757227.094348:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757227.094352:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757227.094361:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10283 00000100:00080000:2.0:1787757227.094370:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757227.094379:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757227.094398:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757227.094432:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588654882176/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757227.094447:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654882048:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757227.094454:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757227.094467:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757227.094470:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757227.094477:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10283 00000100:00080000:2.0:1787757227.094482:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757227.094489:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757227.094499:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757227.094519:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588654882304/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757227.094533:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757227.094538:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757227.094541:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757227.094546:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10283 00000100:00080000:2.0:1787757227.094551:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757227.094555:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757227.094562:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787757227.094570:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757227.094582:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588654882432/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757227.094598:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757227.094603:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757227.094605:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757227.094609:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10283 00000100:00080000:2.0:1787757227.094613:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757227.094618:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757227.094625:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757227.094645:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757227/real 1787757227] req@ffff8c05b2e70a80 x1874588654882048/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757227.094646:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588654882560/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757227.094709:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757227.094715:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757227.094718:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757227.094727:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10283 00000100:00080000:2.0:1787757227.094731:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757227.094735:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757227.094740:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757227.094746:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588654882048/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757227.094754:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588654882688/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757227.094799:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757227.094803:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757227.094805:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787757227.094813:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757227.094825:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10283 00000100:00080000:3.0:1787757227.094825:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787757227.094833:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787757227.094843:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654882048:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757227.094844:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757227.094856:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757227.094933:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588654882816/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757227.094947:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757227.094952:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757227.094954:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757227.094961:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10283 00000100:00080000:2.0:1787757227.094967:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787757227.094972:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654882176:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757227.094978:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757227.094988:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757227.095011:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588654882944/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787757227.095046:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757227.095082:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757227/real 1787757227] req@ffff8c05b2e72300 x1874588654882176/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757227.095112:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588654882176/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757227.095141:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757227.095148:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757227.095156:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654882176:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757227.095170:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654882304:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757227.095224:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757227.095243:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757227/real 1787757227] req@ffff8c05b2e71180 x1874588654882304/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757227.095269:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588654882304/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757227.095280:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757227.095309:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757227.095315:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654882304:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757227.095325:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654882432:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757227.095359:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757227.095381:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757227/real 1787757227] req@ffff8c05b2e72a00 x1874588654882432/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757227.095406:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588654882432/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757227.095424:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757227.095431:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757227.095438:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654882432:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757227.095450:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654882560:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757227.095489:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757227.095505:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757227/real 1787757227] req@ffff8c05b2e70e00 x1874588654882560/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757227.095520:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588654882560/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757227.095532:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757227.095536:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757227.095540:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654882560:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757227.095548:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654882688:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757227.095569:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757227.095585:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757227/real 1787757227] req@ffff8c05b2e72680 x1874588654882688/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757227.095600:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588654882688/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757227.095619:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757227.095623:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757227.095627:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654882688:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757227.095633:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654882816:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757227.095652:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757227.095662:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757227/real 1787757227] req@ffff8c05b2e72d80 x1874588654882816/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757227.095676:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588654882816/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757227.095686:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757227.095689:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757227.095693:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654882816:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757227.095720:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654882944:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757227.095754:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757227.095776:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757227/real 1787757227] req@ffff8c05b2e73100 x1874588654882944/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757227.095802:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588654882944/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757227.095820:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757227.095826:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757227.095834:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654882944:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757227.222038:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757227.670261:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757228.246038:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757228.694157:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757229.270043:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757229.718136:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757230.293984:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757230.742127:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757231.318049:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757231.766313:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757232.214035:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757232.214048:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787757232.214054:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757232.214066:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10288 00000100:00080000:2.0:1787757232.214079:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757232.214088:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757232.214137:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588654883072/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757232.214191:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757232.214207:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757232.214209:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757232.214214:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10288 00000100:00080000:2.0:1787757232.214217:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757232.214239:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757232.214248:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757232.214259:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588654883200/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757232.214267:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757232.214270:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757232.214271:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757232.214274:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10288 00000100:00080000:2.0:1787757232.214277:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757232.214282:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757232.214284:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654883072:192.168.201.107@tcp:250:kworker.0 00000008:00100000:2.0:1787757232.214286:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757232.214295:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588654883328/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757232.214302:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757232.214305:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757232.214306:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757232.214309:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10288 00000100:00080000:2.0:1787757232.214311:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757232.214319:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757232.214323:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757232.214330:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588654883456/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757232.214337:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757232.214339:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757232.214340:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757232.214342:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10288 00000100:00080000:2.0:1787757232.214345:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787757232.214348:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757232.214351:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757232.214356:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757232.214363:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588654883584/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757232.214374:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757232.214376:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757232.214378:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757232.214381:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10288 00000100:00080000:2.0:1787757232.214383:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757232.214386:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757232.214390:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757232/real 1787757232] req@ffff8c05b2e70000 x1874588654883072/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787757232.214391:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757232.214405:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588654883712/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757232.214412:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588654883072/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757232.214415:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757232.214423:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787757232.214426:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757232.214432:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10288 00000100:00080000:3.0:1787757232.214434:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757232.214435:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757232.214441:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787757232.214442:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757232.214447:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757232.214448:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654883072:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787757232.214457:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588654883840/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757232.214458:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654883200:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757232.214472:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757232.214474:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757232.214476:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757232.214480:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10288 00000100:00080000:2.0:1787757232.214484:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757232.214487:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757232.214490:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787757232.214494:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757232.214498:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588654883968/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757232.214508:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757232/real 1787757232] req@ffff8c05b2e70700 x1874588654883200/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757232.214518:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588654883200/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757232.214527:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757232.214531:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757232.214535:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654883200:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757232.214558:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654883328:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757232.214581:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757232.214594:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757232/real 1787757232] req@ffff8c05b2e73480 x1874588654883328/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757232.214607:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588654883328/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757232.214616:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757232.214620:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757232.214623:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654883328:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757232.214630:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654883456:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757232.214652:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757232.214664:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757232/real 1787757232] req@ffff8c05b2e71500 x1874588654883456/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757232.214677:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588654883456/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757232.214686:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757232.214690:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757232.214693:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654883456:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757232.214700:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654883584:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757232.214730:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757232.214742:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757232/real 1787757232] req@ffff8c05b2e73b80 x1874588654883584/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757232.214756:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588654883584/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757232.214762:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757232.214764:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757232.214766:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654883584:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757232.214769:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654883712:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757232.214783:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757232.214789:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757232/real 1787757232] req@ffff8c05b2e71c00 x1874588654883712/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757232.214796:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588654883712/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757232.214812:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757232.214814:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757232.214815:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654883712:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757232.214819:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654883840:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757232.214828:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757232.214834:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757232/real 1787757232] req@ffff8c05b2e71880 x1874588654883840/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757232.214841:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588654883840/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757232.214850:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757232.214868:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757232.214882:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654883840:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757232.214885:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654883968:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757232.214951:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757232.214967:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757232/real 1787757232] req@ffff8c05b2e70380 x1874588654883968/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757232.215000:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588654883968/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757232.215019:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757232.215026:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757232.215032:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654883968:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757232.342020:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757232.790171:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757233.365984:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757233.814287:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757234.390047:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757234.838061:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757235.414018:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757235.862229:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757236.438038:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757236.886140:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757237.333988:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757237.333999:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787757237.334004:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757237.334014:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10293 00000100:00080000:2.0:1787757237.334025:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757237.334033:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757237.334076:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73800 x1874588654884096/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757237.334141:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757237.334148:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757237.334151:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757237.334158:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10293 00000100:00080000:2.0:1787757237.334164:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757237.334172:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757237.334184:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757237.334205:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588654884224/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757237.334220:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787757237.334225:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654884096:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757237.334226:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757237.334231:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757237.334243:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10293 00000100:00080000:2.0:1787757237.334247:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757237.334252:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757237.334259:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757237.334273:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588654884352/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757237.334283:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757237.334287:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757237.334290:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757237.334294:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10293 00000100:00080000:2.0:1787757237.334297:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757237.334301:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757237.334306:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757237.334322:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588654884480/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757237.334328:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757237.334341:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757237.334346:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757237.334349:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757237.334356:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10293 00000100:00080000:2.0:1787757237.334361:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757237.334366:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757237.334373:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757237.334391:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757237/real 1787757237] req@ffff8c05b2e73800 x1874588654884096/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757237.334392:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588654884608/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757237.334439:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757237.334443:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757237.334446:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757237.334449:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73800 x1874588654884096/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757237.334456:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10293 00000100:00080000:2.0:1787757237.334468:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757237.334474:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787757237.334476:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787757237.334480:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787757237.334485:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757237.334493:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654884096:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787757237.334501:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588654884736/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757237.334512:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757237.334516:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787757237.334518:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757237.334525:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654884224:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757237.334527:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10293 00000100:00080000:2.0:1787757237.334537:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757237.334543:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757237.334549:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757237.334562:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588654884864/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757237.334568:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757237.334587:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757237.334591:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757237.334594:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757237.334598:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757237/real 1787757237] req@ffff8c05b2e70380 x1874588654884224/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757237.334605:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10293 00000100:00080000:2.0:1787757237.334616:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757237.334622:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757237.334627:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757237.334628:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588654884224/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757237.334643:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787757237.334646:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588654884992/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757237.334652:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757237.334679:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654884224:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757237.334696:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654884352:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757237.334725:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757237.334738:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757237/real 1787757237] req@ffff8c05b2e71880 x1874588654884352/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757237.334751:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588654884352/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757237.334759:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757237.334762:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757237.334766:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654884352:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757237.334771:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654884480:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757237.334788:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757237.334798:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757237/real 1787757237] req@ffff8c05b2e71c00 x1874588654884480/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757237.334810:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588654884480/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757237.334818:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757237.334821:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757237.334824:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654884480:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757237.334829:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654884608:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757237.334845:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757237.334855:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757237/real 1787757237] req@ffff8c05b2e73b80 x1874588654884608/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757237.334867:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588654884608/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757237.334940:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757237.334945:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757237.334950:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654884608:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757237.334957:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654884736:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757237.334979:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757237.335009:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757237/real 1787757237] req@ffff8c05b2e71500 x1874588654884736/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757237.335023:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588654884736/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757237.335032:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757237.335035:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757237.335038:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654884736:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757237.335058:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654884864:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757237.335078:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757237.335089:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757237/real 1787757237] req@ffff8c05b2e73480 x1874588654884864/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757237.335101:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588654884864/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757237.335133:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757237.335137:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757237.335142:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654884864:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757237.335149:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654884992:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757237.335172:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757237.335184:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757237/real 1787757237] req@ffff8c05b2e70700 x1874588654884992/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757237.335197:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588654884992/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757237.335208:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757237.335213:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757237.335219:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654884992:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757237.462034:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757237.910294:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757238.486034:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757238.934170:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757239.510005:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757239.958072:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757240.534019:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757240.982176:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757241.557999:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757242.006201:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757242.453988:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757242.453999:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787757242.454004:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757242.454030:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10298 00000100:00080000:2.0:1787757242.454042:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757242.454051:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757242.454099:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588654885120/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757242.454129:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757242.454132:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757242.454133:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757242.454137:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10298 00000100:00080000:2.0:1787757242.454139:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757242.454141:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757242.454151:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757242.454161:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588654885248/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757242.454176:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757242.454181:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757242.454186:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757242.454187:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654885120:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757242.454193:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10298 00000100:00080000:2.0:1787757242.454207:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757242.454212:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757242.454216:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757242.454231:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588654885376/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757242.454239:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757242.454242:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757242.454244:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757242.454247:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10298 00000100:00080000:2.0:1787757242.454249:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757242.454252:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757242.454257:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757242.454266:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588654885504/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757242.454274:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787757242.454275:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757242.454285:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757242.454287:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757242.454292:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10298 00000100:00080000:2.0:1787757242.454297:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757242.454303:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757242.454308:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757242.454318:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588654885632/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757242.454326:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787757242.454328:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757242/real 1787757242] req@ffff8c05b2e70000 x1874588654885120/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757242.454334:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757242.454338:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757242.454355:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10298 00000100:00080000:2.0:1787757242.454361:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757242.454364:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757242.454367:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757242.454379:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588654885120/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757242.454382:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588654885760/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757242.454449:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757242.454451:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757242.454456:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787757242.454460:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787757242.454461:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654885120:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757242.454464:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757242.454475:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10298 00000100:00080000:2.0:1787757242.454480:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757242.454484:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757242.454489:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757242.454499:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654885248:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757242.454503:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588654885888/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757242.454534:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757242.454538:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757242.454540:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787757242.454548:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757242.454553:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10298 00000100:00080000:2.0:1787757242.454560:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757242.454565:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757242.454572:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757242/real 1787757242] req@ffff8c05b2e73100 x1874588654885248/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787757242.454574:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757242.454597:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588654886016/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757242.454601:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588654885248/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757242.454660:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757242.454665:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757242.454669:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654885248:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757242.454678:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654885376:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757242.454702:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757242.454715:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757242/real 1787757242] req@ffff8c05b2e72d80 x1874588654885376/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757242.454729:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588654885376/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757242.454737:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757242.454740:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757242.454742:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654885376:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757242.454747:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654885504:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757242.454764:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757242.454771:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757242/real 1787757242] req@ffff8c05b2e72680 x1874588654885504/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757242.454784:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588654885504/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757242.454789:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757242.454791:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757242.454793:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654885504:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757242.454797:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654885632:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757242.454807:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757242.454813:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757242/real 1787757242] req@ffff8c05b2e70e00 x1874588654885632/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757242.454820:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588654885632/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757242.454825:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757242.454827:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757242.454829:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654885632:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757242.454833:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654885760:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757242.454843:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757242.454849:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757242/real 1787757242] req@ffff8c05b2e72a00 x1874588654885760/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757242.454856:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588654885760/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757242.454866:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757242.454868:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757242.454870:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654885760:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757242.454909:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654885888:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757242.454922:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757242.454932:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757242/real 1787757242] req@ffff8c05b2e71180 x1874588654885888/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757242.454944:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588654885888/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757242.454952:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757242.454955:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757242.454959:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654885888:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757242.454966:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654886016:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757242.454986:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757242.454995:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757242/real 1787757242] req@ffff8c05b2e72300 x1874588654886016/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757242.455003:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588654886016/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757242.455009:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757242.455011:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757242.455029:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654886016:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757242.582058:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757243.030239:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757243.606070:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757244.054281:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757244.629970:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757245.078174:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757245.654051:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757246.102298:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757246.678023:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757247.126197:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757247.574097:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757247.574113:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787757247.574120:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757247.574136:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10303 00000100:00080000:0.0:1787757247.574152:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757247.574165:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757247.574224:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588654886144/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757247.574278:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757247.574283:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757247.574286:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757247.574291:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10303 00000100:00080000:0.0:1787757247.574297:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757247.574303:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757247.574314:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757247.574334:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588654886272/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757247.574351:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757247.574356:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757247.574363:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757247.574365:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654886144:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757247.574371:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10303 00000100:00080000:0.0:1787757247.574380:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757247.574389:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757247.574395:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757247.574413:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588654886400/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757247.574427:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757247.574432:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757247.574434:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757247.574439:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10303 00000100:00080000:0.0:1787757247.574447:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757247.574451:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757247.574457:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757247.574471:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588654886528/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757247.574483:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787757247.574491:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757247.574492:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757247.574501:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757247.574508:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10303 00000100:00080000:0.0:1787757247.574514:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757247.574520:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757247.574527:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757247.574546:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588654886656/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757247.574562:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757247/real 1787757247] req@ffff8c05a9f76a00 x1874588654886144/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757247.574572:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757247.574592:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757247.574599:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757247.574610:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588654886144/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757247.574613:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10303 00000100:00080000:0.0:1787757247.574640:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757247.574646:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757247.574656:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:3.0:1787757247.574659:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757247.574673:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654886144:192.168.201.107@tcp:250:kworker.0 00000008:00100000:0.0:1787757247.574674:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757247.574706:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588654886784/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757247.574718:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654886272:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757247.574734:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757247.574742:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787757247.574745:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757247.574753:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10303 00000100:00080000:0.0:1787757247.574760:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787757247.574769:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757247.574776:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757247.574787:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757247.574799:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757247/real 1787757247] req@ffff8c05a9f76d80 x1874588654886272/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757247.574810:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588654886912/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757247.574840:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787757247.574843:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588654886272/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757247.574850:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757247.574857:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757247.574921:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757247.574958:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757247.574966:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654886272:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757247.574977:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654886400:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757247.575019:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757247.575026:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10303 00000100:00080000:0.0:1787757247.575042:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787757247.575046:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757247/real 1787757247] req@ffff8c05a9f77b80 x1874588654886400/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757247.575056:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757247.575076:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757247.575082:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588654886400/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757247.575106:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588654887040/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757247.575110:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757247.575126:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757247.575140:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654886400:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757247.575155:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654886528:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757247.575186:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757247.575205:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757247/real 1787757247] req@ffff8c05a9f76680 x1874588654886528/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757247.575221:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588654886528/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757247.575232:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757247.575236:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757247.575240:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654886528:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757247.575247:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654886656:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757247.575265:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757247.575277:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757247/real 1787757247] req@ffff8c05a9f75c00 x1874588654886656/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757247.575290:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588654886656/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757247.575300:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757247.575303:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757247.575308:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654886656:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757247.575330:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654886784:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757247.575350:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757247.575362:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757247/real 1787757247] req@ffff8c05a9f77800 x1874588654886784/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757247.575383:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588654886784/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757247.575399:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757247.575404:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757247.575411:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654886784:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757247.575424:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654886912:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757247.575462:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757247.575485:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757247/real 1787757247] req@ffff8c05a9f74380 x1874588654886912/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757247.575502:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588654886912/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757247.575512:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757247.575527:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757247.575531:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654886912:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757247.575538:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654887040:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757247.575563:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757247.575575:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757247/real 1787757247] req@ffff8c05a9f74a80 x1874588654887040/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757247.575589:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588654887040/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757247.575599:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757247.575602:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757247.575606:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654887040:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757247.702054:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757248.150243:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757248.725973:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757249.174124:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757249.750017:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757250.198255:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757250.773994:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757251.222133:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757251.798006:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757252.246217:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757252.694101:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757252.694114:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787757252.694121:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757252.694134:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10309 00000100:00080000:2.0:1787757252.694151:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757252.694164:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757252.694224:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588654887168/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757252.694272:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757252.694277:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757252.694280:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757252.694286:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10309 00000100:00080000:2.0:1787757252.694292:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757252.694297:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757252.694310:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757252.694331:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588654887296/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757252.694347:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757252.694352:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757252.694355:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757252.694360:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10309 00000100:00080000:2.0:1787757252.694363:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757252.694368:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757252.694374:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757252.694386:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588654887424/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757252.694403:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757252.694409:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757252.694419:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757252.694424:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654887168:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757252.694425:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10309 00000100:00080000:2.0:1787757252.694439:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757252.694452:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757252.694458:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757252.694480:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588654887552/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757252.694492:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757252.694497:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757252.694502:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757252.694507:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10309 00000100:00080000:2.0:1787757252.694513:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757252.694519:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757252.694529:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757252.694543:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588654887680/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757252.694554:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757252.694560:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757252.694562:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757252.694578:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10309 00000800:00000100:3.0:1787757252.694583:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757252.694588:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757252.694596:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757252.694603:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757252.694624:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588654887808/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757252.694636:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757252.694641:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787757252.694645:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757252.694651:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10309 00000100:00080000:2.0:1787757252.694659:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757252.694668:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757252.694675:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757252/real 1787757252] req@ffff8c05b2e70a80 x1874588654887168/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787757252.694679:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757252.694713:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588654887936/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757252.694729:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588654887168/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757252.694730:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757252.694759:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757252.694769:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757252.694793:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10309 00000100:00080000:3.0:1787757252.694798:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757252.694799:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757252.694821:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757252.694828:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654887168:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757252.694861:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757252.694873:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757252.694935:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654887296:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757252.694979:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588654888064/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787757252.694997:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757252.695039:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757252/real 1787757252] req@ffff8c05b2e72300 x1874588654887296/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757252.695070:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588654887296/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757252.695083:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757252.695088:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757252.695093:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654887296:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757252.695133:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654887424:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757252.695159:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757252.695173:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757252/real 1787757252] req@ffff8c05b2e71180 x1874588654887424/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757252.695188:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588654887424/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757252.695199:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757252.695203:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757252.695207:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654887424:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757252.695215:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654887552:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757252.695237:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757252.695248:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757252/real 1787757252] req@ffff8c05b2e72a00 x1874588654887552/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757252.695263:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588654887552/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757252.695281:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757252.695286:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757252.695290:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654887552:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757252.695297:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654887680:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757252.695319:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757252.695337:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757252/real 1787757252] req@ffff8c05b2e70e00 x1874588654887680/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757252.695360:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588654887680/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757252.695377:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757252.695384:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757252.695392:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654887680:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757252.695407:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654887808:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757252.695445:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757252.695461:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757252/real 1787757252] req@ffff8c05b2e72680 x1874588654887808/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757252.695477:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588654887808/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757252.695487:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757252.695492:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757252.695497:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654887808:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757252.695505:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654887936:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757252.695527:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757252.695539:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757252/real 1787757252] req@ffff8c05b2e72d80 x1874588654887936/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757252.695554:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588654887936/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757252.695564:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757252.695567:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757252.695571:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654887936:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757252.695578:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654888064:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757252.695597:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757252.695608:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757252/real 1787757252] req@ffff8c05b2e73100 x1874588654888064/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757252.695623:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588654888064/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757252.695632:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757252.695636:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757252.695639:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654888064:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757252.821969:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757253.270184:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757253.846002:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757254.294184:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757254.870031:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757255.318108:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757255.893974:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757256.342157:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757256.917990:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757257.366072:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757257.813967:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757257.813980:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787757257.813986:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757257.813998:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10314 00000100:00080000:2.0:1787757257.814011:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757257.814020:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757257.814099:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588654888192/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757257.814161:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757257.814166:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757257.814169:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757257.814175:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10314 00000100:00080000:2.0:1787757257.814182:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757257.814189:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757257.814201:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757257.814221:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588654888320/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757257.814237:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757257.814242:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757257.814245:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757257.814249:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10314 00000100:00080000:2.0:1787757257.814254:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757257.814258:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757257.814264:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757257.814279:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654888192:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787757257.814280:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588654888448/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757257.814309:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757257.814313:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757257.814316:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757257.814320:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10314 00000100:00080000:2.0:1787757257.814325:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757257.814330:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757257.814335:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757257.814351:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588654888576/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757257.814364:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757257.814369:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757257.814372:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757257.814377:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10314 00000100:00080000:2.0:1787757257.814384:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757257.814388:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757257.814396:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787757257.814403:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757257.814442:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588654888704/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757257.814466:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757257.814472:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757257.814476:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757257.814485:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757257/real 1787757257] req@ffff8c05b2e70000 x1874588654888192/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757257.814488:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10314 00000100:00080000:2.0:1787757257.814504:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757257.814519:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757257.814529:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588654888192/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787757257.814537:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787757257.814553:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757257.814561:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757257.814567:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654888192:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787757257.814579:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588654888832/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757257.814585:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654888320:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757257.814615:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757257.814620:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787757257.814624:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787757257.814632:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757257.814636:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10314 00000100:00080000:2.0:1787757257.814648:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757257.814657:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757257.814661:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757257/real 1787757257] req@ffff8c05b2e70700 x1874588654888320/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787757257.814668:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757257.814691:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588654888320/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757257.814699:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588654888960/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757257.814732:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757257.814769:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757257.814771:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757257.814786:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654888320:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757257.814788:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757257.814798:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757257.814811:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10314 00000100:00080000:2.0:1787757257.814820:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757257.814831:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757257.814833:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654888448:192.168.201.107@tcp:38:kworker.0 00000008:00100000:2.0:1787757257.814851:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787757257.814925:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757257.814938:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588654889088/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757257.814949:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757257/real 1787757257] req@ffff8c05b2e73480 x1874588654888448/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757257.815000:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588654888448/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757257.815012:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757257.815018:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757257.815023:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654888448:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757257.815032:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654888576:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757257.815056:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757257.815067:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757257/real 1787757257] req@ffff8c05b2e71500 x1874588654888576/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757257.815081:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588654888576/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757257.815092:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757257.815096:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757257.815099:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654888576:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757257.815106:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654888704:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757257.815123:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757257.815134:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757257/real 1787757257] req@ffff8c05b2e73b80 x1874588654888704/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757257.815149:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588654888704/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757257.815159:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757257.815162:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757257.815166:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654888704:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757257.815172:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654888832:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757257.815190:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757257.815207:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757257/real 1787757257] req@ffff8c05b2e71c00 x1874588654888832/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757257.815230:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588654888832/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757257.815246:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757257.815252:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757257.815260:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654888832:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757257.815272:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654888960:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757257.815312:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757257.815330:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757257/real 1787757257] req@ffff8c05b2e71880 x1874588654888960/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757257.815345:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588654888960/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757257.815364:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757257.815367:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757257.815371:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654888960:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757257.815394:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654889088:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757257.815417:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757257.815429:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757257/real 1787757257] req@ffff8c05b2e70380 x1874588654889088/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757257.815443:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588654889088/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757257.815453:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757257.815457:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757257.815460:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654889088:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757257.942018:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757258.390296:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757258.966033:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757259.414113:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757259.989995:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757260.438219:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757261.014006:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757261.462045:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757262.038074:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757262.486326:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757262.934048:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757262.934078:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787757262.934083:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757262.934096:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10319 00000100:00080000:2.0:1787757262.934111:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757262.934120:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757262.934173:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73800 x1874588654889216/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757262.934221:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757262.934225:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757262.934227:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757262.934231:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10319 00000100:00080000:2.0:1787757262.934236:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757262.934240:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757262.934251:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757262.934278:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588654889344/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757262.934299:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757262.934304:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757262.934307:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757262.934313:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10319 00000100:00080000:2.0:1787757262.934322:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787757262.934322:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654889216:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757262.934331:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757262.934339:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757262.934353:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588654889472/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757262.934364:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757262.934368:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757262.934369:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757262.934373:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10319 00000100:00080000:2.0:1787757262.934377:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757262.934381:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757262.934385:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757262.934395:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588654889600/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757262.934410:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757262.934414:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757262.934416:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757262.934419:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10319 00000100:00080000:2.0:1787757262.934422:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757262.934425:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757262.934429:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787757262.934435:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757262.934441:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588654889728/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757262.934466:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757262.934469:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757262.934471:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757262.934475:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10319 00000100:00080000:2.0:1787757262.934479:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757262.934482:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757262.934486:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757262.934496:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588654889856/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757262.934503:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757262.934507:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787757262.934508:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757262.934512:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10319 00000100:00080000:2.0:1787757262.934515:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757262.934519:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757262.934523:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757262.934526:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757262/real 1787757262] req@ffff8c05b2e73800 x1874588654889216/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757262.934536:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588654889984/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757262.934571:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757262.934575:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757262.934577:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757262.934583:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10319 00000100:00100000:3.0:1787757262.934583:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73800 x1874588654889216/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757262.934586:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757262.934593:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757262.934601:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787757262.934606:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757262.934614:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757262.934615:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588654890112/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757262.934626:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654889216:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757262.934637:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654889344:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757262.934691:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757262.934713:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757262/real 1787757262] req@ffff8c05b2e70380 x1874588654889344/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757262.934724:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588654889344/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757262.934732:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757262.934735:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757262.934737:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654889344:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757262.934765:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654889472:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757262.934782:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757262.934790:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757262/real 1787757262] req@ffff8c05b2e71880 x1874588654889472/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757262.934800:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588654889472/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757262.934812:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757262.934815:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757262.934818:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654889472:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757262.934822:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654889600:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757262.934834:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757262.934841:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757262/real 1787757262] req@ffff8c05b2e71c00 x1874588654889600/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757262.934850:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588654889600/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757262.934857:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757262.934859:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757262.934861:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654889600:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757262.934866:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654889728:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757262.934933:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757262.934944:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757262/real 1787757262] req@ffff8c05b2e73b80 x1874588654889728/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757262.934954:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588654889728/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757262.934977:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757262.934979:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757262.934982:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654889728:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757262.934987:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654889856:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757262.935003:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757262.935011:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757262/real 1787757262] req@ffff8c05b2e71500 x1874588654889856/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757262.935020:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588654889856/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757262.935027:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757262.935028:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757262.935031:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654889856:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757262.935035:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654889984:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757262.935047:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757262.935053:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757262/real 1787757262] req@ffff8c05b2e73480 x1874588654889984/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757262.935063:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588654889984/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757262.935069:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757262.935071:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757262.935073:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654889984:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757262.935076:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654890112:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757262.935088:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757262.935096:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757262/real 1787757262] req@ffff8c05b2e70700 x1874588654890112/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757262.935104:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588654890112/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757262.935111:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757262.935112:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757262.935115:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654890112:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757263.061971:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757263.510135:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757264.086011:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757264.534199:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757265.110001:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757265.558143:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757266.133955:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757266.582305:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757267.158015:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757267.606183:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757268.054016:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757268.054026:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787757268.054030:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757268.054040:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10324 00000100:00080000:2.0:1787757268.054049:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757268.054056:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757268.054094:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588654890240/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757268.054120:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757268.054124:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757268.054131:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757268.054134:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10324 00000100:00080000:2.0:1787757268.054138:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757268.054141:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757268.054149:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757268.054162:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588654890368/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757268.054172:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757268.054178:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1787757268.054180:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654890240:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757268.054181:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757268.054192:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10324 00000100:00080000:2.0:1787757268.054196:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757268.054198:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757268.054202:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757268.054213:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588654890496/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757268.054222:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757268.054232:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757268.054234:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757268.054237:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10324 00000100:00080000:2.0:1787757268.054241:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757268.054245:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757268.054250:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757268.054259:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588654890624/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757268.054268:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757268.054271:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757268.054272:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757268.054275:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10324 00000100:00080000:2.0:1787757268.054277:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757268.054281:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757268.054285:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757268.054294:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588654890752/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757268.054302:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757268.054305:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757268.054310:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757268.054310:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654890368:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757268.054321:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10324 00000100:00080000:2.0:1787757268.054325:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757268.054335:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757268.054340:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757268.054353:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654890496:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757268.054356:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588654890880/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757268.054370:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757268.054373:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787757268.054376:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757268.054380:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654890624:192.168.201.107@tcp:8:kworker.0 00000100:00080000:2.0:1787757268.054385:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10324 00000100:00080000:2.0:1787757268.054392:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757268.054397:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757268.054402:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757268.054409:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654890752:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757268.054416:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588654891008/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757268.054426:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757268.054429:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757268.054431:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757268.054435:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654890880:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757268.054438:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10324 00000100:00080000:2.0:1787757268.054448:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757268.054452:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757268.054458:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757268.054465:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654891008:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757268.054472:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588654891136/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757268.054497:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654891136:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757268.181981:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757268.630169:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757269.205974:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757269.654065:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757270.229998:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757270.678217:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787757271.126271:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.107:988 00000400:00000100:1.0:1787757271.126325:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.107@tcp at host 192.168.201.107 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787757271.126352:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787757271.126415:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787757271.126434:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787757271.126452:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787757271.126468:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787757271.126487:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:3.0:1787757271.126491:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757268/real 1787757271] req@ffff8c05b2e73100 x1874588654890368/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787757271.126505:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787757271.126559:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:3.0:1787757271.126587:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588654890368/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757271.126631:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757271.126641:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757271.126650:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654890368:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757271.126667:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757268/real 1787757271] req@ffff8c05b2e72d80 x1874588654890496/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757271.126693:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588654890496/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757271.126706:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757271.126712:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757271.126718:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654890496:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757271.126731:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757268/real 1787757271] req@ffff8c05b2e72680 x1874588654890624/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757271.126751:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588654890624/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757271.126767:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757271.126773:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757271.126779:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654890624:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757271.126790:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757268/real 1787757271] req@ffff8c05b2e70e00 x1874588654890752/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757271.126810:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588654890752/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757271.126823:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757271.126829:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757271.126835:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654890752:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757271.126847:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757268/real 1787757271] req@ffff8c05b2e70000 x1874588654890240/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757271.126870:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588654890240/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757271.126944:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757271.126952:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757271.126959:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654890240:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757271.127007:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757268/real 1787757271] req@ffff8c05b2e72300 x1874588654891136/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757271.127033:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588654891136/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757271.127048:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757271.127055:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757271.127061:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654891136:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757271.127071:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757268/real 1787757271] req@ffff8c05b2e72a00 x1874588654890880/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757271.127093:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588654890880/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757271.127110:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757271.127116:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757271.127121:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654890880:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757271.127133:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757268/real 1787757271] req@ffff8c05b2e71180 x1874588654891008/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757271.127156:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588654891008/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757271.127173:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757271.127179:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757271.127185:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654891008:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757271.254079:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757271.702149:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757272.278017:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757272.726220:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757273.174057:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757273.174069:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787757273.174076:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757273.174088:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10329 00000100:00080000:2.0:1787757273.174103:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757273.174114:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757273.174162:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588654891264/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757273.174204:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757273.174209:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757273.174213:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757273.174220:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10329 00000100:00080000:2.0:1787757273.174225:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757273.174230:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757273.174243:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757273.174246:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654891264:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787757273.174268:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588654891392/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757273.174287:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757273.174292:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757273.174295:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757273.174301:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10329 00000100:00080000:2.0:1787757273.174307:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757273.174313:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757273.174321:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757273.174339:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588654891520/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757273.174354:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787757273.174358:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757273.174361:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757273.174368:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757273.174375:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10329 00000100:00080000:2.0:1787757273.174383:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757273.174389:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757273.174396:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757273.174412:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588654891648/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757273.174425:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757273.174434:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1787757273.174437:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757273/real 1787757273] req@ffff8c05b2e70a80 x1874588654891264/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757273.174439:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757273.174455:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10329 00000100:00080000:2.0:1787757273.174471:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757273.174517:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757273.174530:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757273.174534:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588654891264/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757273.174563:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787757273.174565:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588654891776/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757273.174582:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757273.174602:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654891264:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757273.174605:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757273.174618:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757273.174623:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757273.174632:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10329 00000100:00080000:2.0:1787757273.174645:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757273.174655:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757273.174664:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757273.174681:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654891392:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757273.174694:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588654891904/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757273.174730:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757273.174737:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1787757273.174746:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757273.174747:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757273.174764:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10329 00000100:00080000:2.0:1787757273.174773:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757273.174789:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757273.174791:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757273/real 1787757273] req@ffff8c05b2e71180 x1874588654891392/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787757273.174802:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757273.174831:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588654892032/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757273.174843:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588654891392/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757273.174865:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757273.174921:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757273.174925:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757273.174935:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10329 00000100:00080000:2.0:1787757273.174943:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757273.174947:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757273.174955:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787757273.174966:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000008:00100000:2.0:1787757273.174973:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757273.174980:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654891392:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757273.175001:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654891520:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757273.175010:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588654892160/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757273.175060:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757273.175092:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757273/real 1787757273] req@ffff8c05b2e72a00 x1874588654891520/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757273.175119:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588654891520/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757273.175135:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757273.175142:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757273.175148:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654891520:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757273.175161:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654891648:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757273.175218:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757273.175247:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757273/real 1787757273] req@ffff8c05b2e72300 x1874588654891648/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757273.175272:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588654891648/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757273.175288:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757273.175295:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757273.175302:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654891648:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757273.175313:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654891776:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757273.175352:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757273.175373:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757273/real 1787757273] req@ffff8c05b2e70000 x1874588654891776/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757273.175395:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588654891776/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757273.175411:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757273.175417:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757273.175422:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654891776:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757273.175459:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654891904:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757273.175530:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757273.175556:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757273/real 1787757273] req@ffff8c05b2e70e00 x1874588654891904/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757273.175579:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588654891904/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757273.175597:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757273.175604:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757273.175611:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654891904:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757273.175623:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654892032:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757273.175689:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757273.175714:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757273/real 1787757273] req@ffff8c05b2e72680 x1874588654892032/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757273.175753:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588654892032/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757273.175769:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757273.175775:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757273.175782:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654892032:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757273.175797:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654892160:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757273.175840:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757273.175926:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757273/real 1787757273] req@ffff8c05b2e72d80 x1874588654892160/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757273.175955:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588654892160/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757273.175973:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757273.175982:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757273.175991:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654892160:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757273.302027:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757273.750153:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757274.326042:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757274.774209:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757275.350019:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757275.798123:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757276.374007:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757276.822234:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757277.398022:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757277.846138:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757278.293989:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757278.294003:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787757278.294009:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757278.294023:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10334 00000100:00080000:2.0:1787757278.294038:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757278.294048:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757278.294098:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588654892288/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757278.294137:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757278.294142:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757278.294146:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757278.294155:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10334 00000100:00080000:2.0:1787757278.294163:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757278.294173:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757278.294189:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757278.294200:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654892288:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787757278.294226:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588654892416/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757278.294255:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757278.294261:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757278.294263:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757278.294269:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10334 00000100:00080000:2.0:1787757278.294277:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757278.294283:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:0.0:1787757278.294290:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:2.0:1787757278.294298:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757278.294323:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588654892544/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757278.294338:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757278.294343:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757278.294346:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757278.294350:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10334 00000100:00080000:2.0:1787757278.294356:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757278.294366:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757278/real 1787757278] req@ffff8c05b2e73100 x1874588654892288/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757278.294367:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757278.294380:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757278.294434:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588654892672/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757278.294455:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757278.294460:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1787757278.294466:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588654892288/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757278.294467:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757278.294483:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10334 00000100:00080000:2.0:1787757278.294495:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757278.294509:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757278.294523:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787757278.294540:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787757278.294545:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588654892800/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757278.294558:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787757278.294570:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787757278.294578:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654892288:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757278.294579:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757278.294586:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757278.294600:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10334 00000100:00080000:2.0:1787757278.294605:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757278.294611:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757278.294617:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757278.294634:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654892416:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757278.294635:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588654892928/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757278.294659:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757278.294663:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787757278.294666:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757278.294672:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10334 00000100:00080000:2.0:1787757278.294678:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757278.294684:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:0.0:1787757278.294693:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:2.0:1787757278.294702:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757278.294720:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757278/real 1787757278] req@ffff8c05b2e70700 x1874588654892416/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757278.294738:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588654893056/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757278.294745:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588654892416/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757278.294793:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757278.294800:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757278.294810:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757278.294813:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757278.294832:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787757278.294833:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10334 00000100:00100000:0.0:1787757278.294843:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654892416:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757278.294851:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757278.294858:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654892544:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757278.294869:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:0.0:1787757278.294924:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:2.0:1787757278.294938:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757278.294953:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757278/real 1787757278] req@ffff8c05b2e73480 x1874588654892544/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757278.294971:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588654893184/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757278.294994:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588654892544/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757278.295018:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757278.295027:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757278.295037:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654892544:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757278.295053:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654892672:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757278.295090:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757278.295106:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757278/real 1787757278] req@ffff8c05b2e71500 x1874588654892672/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757278.295121:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588654892672/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757278.295132:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757278.295136:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757278.295140:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654892672:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757278.295148:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654892800:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757278.295168:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757278.295179:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757278/real 1787757278] req@ffff8c05b2e73b80 x1874588654892800/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757278.295193:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588654892800/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757278.295203:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757278.295206:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757278.295210:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654892800:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757278.295234:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654892928:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757278.295255:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757278.295267:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757278/real 1787757278] req@ffff8c05b2e71c00 x1874588654892928/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757278.295281:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588654892928/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757278.295292:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757278.295295:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757278.295299:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654892928:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757278.295305:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654893056:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757278.295324:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757278.295335:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757278/real 1787757278] req@ffff8c05b2e71880 x1874588654893056/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757278.295349:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588654893056/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757278.295358:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757278.295362:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757278.295365:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654893056:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757278.295371:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654893184:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757278.295425:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757278.295440:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757278/real 1787757278] req@ffff8c05b2e70380 x1874588654893184/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757278.295455:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588654893184/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757278.295473:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757278.295477:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757278.295481:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654893184:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757278.421971:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757278.870202:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757279.445999:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757279.894140:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757280.469996:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757280.918262:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757281.494014:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757281.942141:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757282.518005:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757282.966214:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757283.414140:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757283.414154:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787757283.414160:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757283.414173:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10339 00000100:00080000:1.0:1787757283.414187:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757283.414196:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757283.414245:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588654893312/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757283.414289:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757283.414327:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757283.414330:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757283.414336:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10339 00000100:00080000:1.0:1787757283.414341:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757283.414347:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757283.414361:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757283.414389:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654893312:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787757283.414390:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588654893440/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757283.414418:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757283.414423:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757283.414425:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757283.414432:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10339 00000100:00080000:1.0:1787757283.414438:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757283.414443:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757283.414452:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757283.414471:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588654893568/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757283.414485:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757283.414490:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757283.414492:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757283.414497:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10339 00000100:00080000:1.0:1787757283.414502:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787757283.414507:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757283.414512:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757283.414523:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757283.414539:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654893696/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757283.414551:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757283.414556:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757283.414558:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757283.414564:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10339 00000100:00080000:1.0:1787757283.414571:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757283.414576:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757283.414585:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757283/real 1787757283] req@ffff8c05b2d9bb80 x1874588654893312/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787757283.414587:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757283.414613:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654893824/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757283.414626:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588654893312/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757283.414672:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757283.414680:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757283.414683:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757283.414684:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757283.414704:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10339 00000100:00080000:0.0:1787757283.414705:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787757283.414709:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757283.414719:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757283.414727:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654893312:192.168.201.107@tcp:250:kworker.0 00000008:00100000:1.0:1787757283.414729:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757283.414749:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588654893952/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757283.414761:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757283.414765:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787757283.414768:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757283.414772:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10339 00000100:00080000:1.0:1787757283.414776:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757283.414780:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757283.414784:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757283.414801:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654894080/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757283.414807:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654893440:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787757283.414841:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757283.414848:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757283.414852:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757283.414863:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10339 00000800:00000100:0.0:1787757283.414950:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757283.414978:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757283/real 1787757283] req@ffff8c05b2d9aa00 x1874588654893440/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757283.414983:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757283.415001:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757283.415010:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757283.415027:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588654893440/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757283.415043:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654894208/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757283.415047:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757283.415062:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757283.415076:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654893440:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757283.415091:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654893568:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757283.415127:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757283.415146:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757283/real 1787757283] req@ffff8c05b2d9b100 x1874588654893568/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757283.415162:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654893568/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757283.415172:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757283.415176:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757283.415180:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654893568:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757283.415188:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654893696:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757283.415208:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757283.415220:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757283/real 1787757283] req@ffff8c05b2d99880 x1874588654893696/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757283.415238:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654893696/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757283.415254:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757283.415260:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757283.415266:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654893696:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757283.415278:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654893824:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757283.415346:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757283.415372:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757283/real 1787757283] req@ffff8c05b2d98380 x1874588654893824/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757283.415397:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654893824/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757283.415410:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757283.415416:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757283.415423:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654893824:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757283.415436:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654893952:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757283.415478:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757283.415501:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757283/real 1787757283] req@ffff8c05b2d9b480 x1874588654893952/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757283.415526:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654893952/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757283.415537:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757283.415541:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757283.415546:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654893952:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757283.415574:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654894080:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757283.415601:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757283.415615:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757283/real 1787757283] req@ffff8c05b2d98000 x1874588654894080/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757283.415629:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654894080/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757283.415641:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757283.415647:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757283.415654:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654894080:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757283.415665:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654894208:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757283.415701:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757283.415726:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757283/real 1787757283] req@ffff8c05b2d99f80 x1874588654894208/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757283.415751:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654894208/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757283.415768:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757283.415774:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757283.415781:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654894208:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757283.542004:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757283.990155:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757284.566021:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757285.014231:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757285.590049:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757286.038095:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757286.614011:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757287.062378:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757287.637966:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757288.086073:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757288.534199:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757288.534256:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787757288.534263:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757288.534276:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10344 00000100:00080000:2.0:1787757288.534289:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757288.534298:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757288.534344:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73800 x1874588654894336/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757288.534393:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757288.534398:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757288.534400:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757288.534406:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10344 00000100:00080000:2.0:1787757288.534411:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757288.534417:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757288.534428:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757288.534449:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588654894464/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757288.534464:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757288.534469:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757288.534472:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757288.534476:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10344 00000100:00080000:2.0:1787757288.534480:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757288.534486:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757288.534496:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654894336:192.168.201.107@tcp:250:kworker.0 00000008:00100000:2.0:1787757288.534497:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757288.534532:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588654894592/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757288.534549:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757288.534552:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757288.534554:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757288.534557:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10344 00000100:00080000:2.0:1787757288.534559:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757288.534562:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757288.534568:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757288.534581:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588654894720/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757288.534589:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757288.534593:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757288.534594:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757288.534598:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10344 00000100:00080000:2.0:1787757288.534601:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757288.534604:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757288.534609:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787757288.534615:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757288.534623:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588654894848/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757288.534634:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757288.534637:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757288.534639:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757288.534643:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10344 00000100:00080000:2.0:1787757288.534647:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757288.534652:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757288.534659:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757288.534673:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588654894976/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757288.534677:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757288/real 1787757288] req@ffff8c05b2e73800 x1874588654894336/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757288.534713:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757288.534721:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787757288.534725:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757288.534726:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73800 x1874588654894336/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757288.534735:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10344 00000100:00080000:2.0:1787757288.534746:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757288.534752:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757288.534753:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787757288.534766:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000008:00100000:2.0:1787757288.534767:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757288.534775:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654894336:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787757288.534788:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654894464:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757288.534788:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588654895104/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757288.534807:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757288.534811:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757288.534813:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757288.534818:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10344 00000100:00080000:2.0:1787757288.534822:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787757288.534824:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757288.534831:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757288.534840:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757288.534844:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757288/real 1787757288] req@ffff8c05b2e70380 x1874588654894464/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757288.534867:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588654894464/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757288.534867:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588654895232/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757288.534955:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757288.534961:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757288.534968:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654894464:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757288.535009:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654894592:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757288.535064:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757288.535080:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757288/real 1787757288] req@ffff8c05b2e71880 x1874588654894592/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757288.535092:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588654894592/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757288.535100:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757288.535103:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757288.535106:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654894592:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757288.535111:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654894720:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757288.535146:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757288.535158:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757288/real 1787757288] req@ffff8c05b2e71c00 x1874588654894720/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757288.535174:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588654894720/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757288.535184:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757288.535187:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757288.535191:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654894720:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757288.535198:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654894848:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757288.535270:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757288.535281:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757288/real 1787757288] req@ffff8c05b2e73b80 x1874588654894848/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757288.535292:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588654894848/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757288.535300:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757288.535303:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757288.535306:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654894848:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757288.535311:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654894976:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757288.535325:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757288.535332:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757288/real 1787757288] req@ffff8c05b2e71500 x1874588654894976/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757288.535355:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588654894976/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757288.535370:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757288.535373:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757288.535376:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654894976:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757288.535381:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654895104:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757288.535434:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757288.535456:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757288/real 1787757288] req@ffff8c05b2e73480 x1874588654895104/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757288.535482:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588654895104/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757288.535499:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757288.535506:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757288.535513:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654895104:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757288.535526:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654895232:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757288.535571:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757288.535582:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757288/real 1787757288] req@ffff8c05b2e70700 x1874588654895232/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757288.535593:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588654895232/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757288.535600:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757288.535603:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757288.535607:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654895232:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757288.661971:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757289.110259:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757289.685940:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757290.134132:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757290.710075:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757291.158352:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757291.734079:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757292.182201:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757292.758028:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757293.206268:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757293.654089:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757293.654102:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757293.654107:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757293.654147:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10350 00000100:00080000:3.0:1787757293.654161:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757293.654170:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757293.654217:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588654895360/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757293.654261:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757293.654266:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757293.654269:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757293.654274:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10350 00000100:00080000:3.0:1787757293.654279:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757293.654283:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757293.654294:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757293.654309:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588654895488/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757293.654323:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757293.654328:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757293.654331:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757293.654336:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10350 00000100:00080000:3.0:1787757293.654339:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757293.654343:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757293.654349:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757293.654360:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588654895616/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757293.654381:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654895360:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757293.654388:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757293.654395:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757293.654398:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757293.654403:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10350 00000100:00080000:3.0:1787757293.654407:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757293.654414:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757293.654424:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757293.654447:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588654895744/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757293.654467:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787757293.654476:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757293.654483:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757293.654490:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757293.654501:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10350 00000100:00080000:3.0:1787757293.654510:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757293.654519:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757293.654531:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757293.654539:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757293/real 1787757293] req@ffff8c05a9f79880 x1874588654895360/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757293.654559:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588654895872/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757293.654570:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588654895360/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757293.654582:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757293.654599:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757293.654603:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757293.654605:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757293.654613:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10350 00000100:00080000:0.0:1787757293.654618:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787757293.654625:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757293.654632:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654895360:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757293.654640:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757293.654651:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757293.654654:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654895488:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757293.654671:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588654896000/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757293.654686:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787757293.654695:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757293.654697:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757293.654705:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757293.654713:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10350 00000100:00080000:3.0:1787757293.654719:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757293.654726:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757293/real 1787757293] req@ffff8c05a9f7a300 x1874588654895488/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757293.654728:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757293.654742:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757293.654766:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588654895488/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757293.654772:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588654896128/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757293.654832:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757293.654834:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757293.654837:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757293.654844:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757293.654855:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787757293.654855:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10350 00000100:00080000:3.0:1787757293.654865:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757293.654914:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757293.654922:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757293.654937:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588654896256/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757293.654961:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654895488:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757293.655014:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654895616:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757293.655053:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757293.655070:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757293/real 1787757293] req@ffff8c05a9f7aa00 x1874588654895616/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757293.655086:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588654895616/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757293.655098:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757293.655102:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757293.655106:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654895616:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757293.655143:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654895744:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757293.655180:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757293.655203:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757293/real 1787757293] req@ffff8c05a9f78700 x1874588654895744/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757293.655228:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588654895744/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757293.655247:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757293.655253:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757293.655261:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654895744:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757293.655273:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654895872:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757293.655304:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757293.655319:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757293/real 1787757293] req@ffff8c05a9f7b480 x1874588654895872/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757293.655343:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588654895872/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757293.655354:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757293.655358:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757293.655361:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654895872:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757293.655368:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654896000:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757293.655386:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757293.655397:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757293/real 1787757293] req@ffff8c05a9f78a80 x1874588654896000/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757293.655411:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588654896000/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757293.655420:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757293.655424:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757293.655427:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654896000:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757293.655434:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654896128:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757293.655452:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757293.655463:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757293/real 1787757293] req@ffff8c05a9f7a680 x1874588654896128/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757293.655476:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588654896128/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757293.655493:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757293.655497:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757293.655501:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654896128:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757293.655507:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654896256:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757293.655524:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757293.655534:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757293/real 1787757293] req@ffff8c05a9f78380 x1874588654896256/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757293.655547:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588654896256/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757293.655557:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757293.655561:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757293.655564:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654896256:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757293.781976:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757294.230124:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757294.806036:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757295.254153:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757295.829953:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757296.278177:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757296.854027:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757297.302257:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757297.878003:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757298.326093:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757298.774048:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757298.774060:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787757298.774066:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757298.774079:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10355 00000100:00080000:0.0:1787757298.774092:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757298.774101:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757298.774150:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588654896384/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757298.774191:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757298.774197:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757298.774200:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757298.774205:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10355 00000100:00080000:0.0:1787757298.774210:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757298.774215:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757298.774227:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757298.774245:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588654896512/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757298.774264:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654896384:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757298.774266:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757298.774279:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757298.774285:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757298.774292:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10355 00000100:00080000:0.0:1787757298.774296:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757298.774300:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757298.774307:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757298.774322:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588654896640/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757298.774335:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757298.774341:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757298.774343:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757298.774348:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10355 00000100:00080000:0.0:1787757298.774354:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757298.774358:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757298.774366:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757298.774382:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588654896768/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787757298.774392:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757298.774411:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757298.774419:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757298.774423:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757298.774434:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10355 00000100:00080000:0.0:1787757298.774442:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757298.774451:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757298.774462:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757298.774484:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757298/real 1787757298] req@ffff8c05a9f74a80 x1874588654896384/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757298.774492:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588654896896/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757298.774533:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757298.774541:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1787757298.774543:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588654896384/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757298.774548:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757298.774564:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10355 00000100:00080000:0.0:1787757298.774576:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757298.774576:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757298.774587:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:3.0:1787757298.774596:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000008:00100000:0.0:1787757298.774598:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757298.774614:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654896384:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787757298.774624:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588654897024/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757298.774639:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757298.774643:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787757298.774647:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757298.774651:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10355 00000100:00080000:0.0:1787757298.774655:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757298.774659:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757298.774666:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757298.774670:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654896512:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757298.774685:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588654897152/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757298.774702:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757298.774706:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757298.774709:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757298.774715:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10355 00000100:00080000:0.0:1787757298.774723:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787757298.774732:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757298.774735:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757298.774750:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757298.774764:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757298/real 1787757298] req@ffff8c05a9f74380 x1874588654896512/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757298.774781:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588654897280/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757298.774795:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588654896512/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757298.774835:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757298.774841:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757298.774849:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654896512:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757298.774859:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654896640:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757298.774923:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757298.774940:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757298/real 1787757298] req@ffff8c05a9f77800 x1874588654896640/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757298.774955:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588654896640/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757298.774966:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757298.774970:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757298.774974:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654896640:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757298.774982:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654896768:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757298.775006:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757298.775048:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757298/real 1787757298] req@ffff8c05a9f75c00 x1874588654896768/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757298.775073:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588654896768/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757298.775090:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757298.775097:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757298.775106:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654896768:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757298.775120:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654896896:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757298.775156:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757298.775172:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757298/real 1787757298] req@ffff8c05a9f76680 x1874588654896896/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757298.775195:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588654896896/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757298.775212:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757298.775218:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757298.775224:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654896896:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757298.775237:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654897024:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757298.775278:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757298.775300:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757298/real 1787757298] req@ffff8c05a9f77b80 x1874588654897024/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757298.775323:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588654897024/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757298.775340:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757298.775347:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757298.775354:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654897024:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757298.775392:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654897152:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757298.775425:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757298.775441:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757298/real 1787757298] req@ffff8c05a9f76d80 x1874588654897152/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757298.775457:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588654897152/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757298.775467:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757298.775471:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757298.775475:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654897152:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757298.775482:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654897280:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757298.775501:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757298.775512:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757298/real 1787757298] req@ffff8c05a9f76a00 x1874588654897280/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757298.775525:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588654897280/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757298.775536:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757298.775539:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757298.775542:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654897280:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757298.902066:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757299.350249:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757299.925976:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757300.374175:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757300.950063:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757301.398224:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757301.974052:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757302.422114:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757302.998004:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757303.446247:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757303.894126:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757303.894139:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757303.894145:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757303.894159:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10360 00000100:00080000:3.0:1787757303.894185:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757303.894215:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757303.894290:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588654897408/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757303.894341:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757303.894348:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757303.894351:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757303.894360:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10360 00000100:00080000:3.0:1787757303.894367:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757303.894375:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757303.894391:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757303.894417:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588654897536/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757303.894430:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654897408:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757303.894464:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757303.894472:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757303.894477:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757303.894488:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10360 00000100:00080000:3.0:1787757303.894496:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757303.894506:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757303.894517:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787757303.894518:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757303.894555:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588654897664/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757303.894610:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757303.894620:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787757303.894622:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757303/real 1787757303] req@ffff8c05a9f78380 x1874588654897408/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757303.894625:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757303.894643:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10360 00000100:00080000:3.0:1787757303.894660:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757303.894672:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757303.894683:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588654897408/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787757303.894684:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787757303.894719:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757303.894730:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757303.894734:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588654897792/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757303.894738:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654897408:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757303.894768:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757303.894776:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1787757303.894782:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654897536:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757303.894784:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757303.894801:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10360 00000100:00080000:3.0:1787757303.894810:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757303.894818:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757303.894829:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787757303.894830:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757303.894853:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757303/real 1787757303] req@ffff8c05a9f7a680 x1874588654897536/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757303.894862:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588654897920/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757303.894928:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588654897536/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757303.894968:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757303.894985:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757303.894989:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654897536:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757303.895001:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654897664:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757303.895031:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757303.895056:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757303/real 1787757303] req@ffff8c05a9f78a80 x1874588654897664/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757303.895070:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588654897664/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757303.895079:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757303.895083:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757303.895087:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654897664:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757303.895094:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654897792:192.168.201.107@tcp:8:kworker.0 00000100:00080000:3.0:1787757303.895129:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787757303.895131:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757303.895142:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757303.895147:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757303.895151:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757303/real 1787757303] req@ffff8c05a9f7b480 x1874588654897792/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757303.895162:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10360 00000100:00100000:0.0:1787757303.895176:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588654897792/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757303.895179:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757303.895214:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787757303.895215:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757303.895231:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000008:00100000:3.0:1787757303.895236:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757303.895241:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654897792:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757303.895284:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588654898048/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757303.895300:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654897920:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757303.895315:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757303.895325:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757303.895329:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757303.895340:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10360 00000100:00080000:3.0:1787757303.895349:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787757303.895358:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757303.895361:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757303.895374:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757303.895390:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757303/real 1787757303] req@ffff8c05a9f78700 x1874588654897920/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757303.895403:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588654898176/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757303.895418:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588654897920/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757303.895439:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757303.895443:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757303.895454:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757303.895456:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787757303.895464:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757303.895466:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654897920:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757303.895481:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10360 00000100:00080000:3.0:1787757303.895492:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757303.895498:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654898048:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757303.895507:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757303.895521:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787757303.895541:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757303.895550:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588654898304/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757303.895568:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757303/real 1787757303] req@ffff8c05a9f7aa00 x1874588654898048/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757303.895654:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588654898048/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757303.895668:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757303.895673:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757303.895677:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654898048:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757303.895684:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654898176:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757303.895721:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757303.895735:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757303/real 1787757303] req@ffff8c05a9f7a300 x1874588654898176/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757303.895748:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588654898176/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757303.895757:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757303.895761:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757303.895765:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654898176:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757303.895787:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654898304:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757303.895804:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757303.895815:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757303/real 1787757303] req@ffff8c05a9f79880 x1874588654898304/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757303.895827:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588654898304/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757303.895843:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757303.895847:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757303.895850:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654898304:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757304.022051:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757304.470168:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757305.046061:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757305.494191:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757306.070024:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757306.518141:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757307.094046:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757307.542142:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757308.117987:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757308.566177:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757309.013976:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757309.013987:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757309.013992:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757309.014003:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10365 00000100:00080000:3.0:1787757309.014016:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757309.014024:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757309.014067:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588654898432/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757309.014101:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757309.014105:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757309.014107:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757309.014113:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10365 00000100:00080000:3.0:1787757309.014117:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757309.014122:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757309.014133:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757309.014149:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588654898560/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757309.014162:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757309.014167:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757309.014170:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757309.014176:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10365 00000100:00080000:3.0:1787757309.014181:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757309.014185:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654898432:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757309.014191:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757309.014200:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757309.014219:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588654898688/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757309.014235:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757309.014240:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757309.014242:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757309.014248:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10365 00000100:00080000:3.0:1787757309.014254:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787757309.014258:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757309.014266:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757309.014274:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757309.014292:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588654898816/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757309.014303:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757309/real 1787757309] req@ffff8c05a9f7ad80 x1874588654898432/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757309.014309:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757309.014322:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757309.014327:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757309.014335:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588654898432/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757309.014337:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10365 00000100:00080000:3.0:1787757309.014347:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757309.014352:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757309.014357:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787757309.014361:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757309.014368:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654898432:192.168.201.107@tcp:250:kworker.0 00000008:00100000:3.0:1787757309.014370:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757309.014377:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654898560:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757309.014392:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588654898944/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757309.014401:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757309.014415:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787757309.014416:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757309/real 1787757309] req@ffff8c05a9f7bb80 x1874588654898560/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757309.014428:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757309.014435:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757309.014439:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588654898560/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757309.014446:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10365 00000100:00080000:3.0:1787757309.014455:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757309.014461:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787757309.014466:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787757309.014470:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787757309.014472:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757309.014480:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654898560:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757309.014493:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588654899072/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757309.014506:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654898688:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757309.014514:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757309.014520:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757309.014524:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787757309.014530:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757309.014536:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10365 00000100:00080000:3.0:1787757309.014544:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757309.014547:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757309/real 1787757309] req@ffff8c05a9f79180 x1874588654898688/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757309.014558:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588654898688/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757309.014570:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757309.014570:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787757309.014579:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757309.014584:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654898688:192.168.201.107@tcp:38:kworker.0 00000008:00100000:3.0:1787757309.014585:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757309.014590:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654898816:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757309.014610:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588654899200/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757309.014613:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757309.014625:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757309/real 1787757309] req@ffff8c05a9f79f80 x1874588654898816/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757309.014632:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787757309.014640:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588654898816/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757309.014640:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757309.014647:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757309.014654:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757309.014659:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10365 00000100:00080000:0.0:1787757309.014662:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757309.014669:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654898816:192.168.201.107@tcp:8:kworker.0 00000100:00080000:3.0:1787757309.014670:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757309.014676:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654898944:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757309.014681:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757309.014691:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787757309.014698:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757309.014710:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757309/real 1787757309] req@ffff8c05a9f78e00 x1874588654898944/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757309.014716:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588654899328/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757309.014728:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588654898944/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757309.014745:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757309.014749:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757309.014753:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654898944:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757309.014766:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654899072:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757309.014783:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757309.014791:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757309/real 1787757309] req@ffff8c05a9f79500 x1874588654899072/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757309.014806:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588654899072/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757309.014816:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757309.014820:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757309.014824:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654899072:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757309.014833:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654899200:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757309.014916:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757309.014935:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757309/real 1787757309] req@ffff8c05a9f7b100 x1874588654899200/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757309.014954:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588654899200/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757309.014966:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757309.014972:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757309.014977:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654899200:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757309.014987:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654899328:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757309.015018:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757309.015040:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757309/real 1787757309] req@ffff8c05a9f7b800 x1874588654899328/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757309.015055:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588654899328/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757309.015062:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757309.015067:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757309.015073:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654899328:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757309.142045:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757309.590140:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757310.165977:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757310.614111:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757311.190006:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757311.638305:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757312.213971:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757312.662048:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757313.238005:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757313.686144:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757314.134086:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757314.134102:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787757314.134109:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757314.134125:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10370 00000100:00080000:1.0:1787757314.134142:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757314.134155:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757314.134212:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588654899456/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757314.134261:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757314.134267:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757314.134271:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757314.134279:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10370 00000100:00080000:1.0:1787757314.134286:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757314.134293:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757314.134307:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757314.134334:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588654899584/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757314.134358:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787757314.134362:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654899456:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757314.134367:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757314.134376:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757314.134387:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10370 00000100:00080000:1.0:1787757314.134393:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757314.134402:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757314.134413:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757314.134440:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654899712/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757314.134462:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787757314.134464:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757314.134475:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757314.134481:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757314.134493:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10370 00000100:00080000:1.0:1787757314.134515:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757314.134539:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757314.134549:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757314/real 1787757314] req@ffff8c05b2d98700 x1874588654899456/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787757314.134555:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757314.134589:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654899840/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757314.134598:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654899456/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757314.134641:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757314.134645:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757314.134659:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757314.134661:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787757314.134669:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757314.134674:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654899456:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757314.134688:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10370 00000100:00080000:1.0:1787757314.134698:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757314.134707:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757314.134717:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654899584:192.168.201.107@tcp:38:kworker.0 00000008:00100000:1.0:1787757314.134722:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757314.134796:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588654899968/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757314.134798:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757314.134822:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757314.134828:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757314.134832:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757314.134840:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10370 00000100:00100000:0.0:1787757314.134842:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757314/real 1787757314] req@ffff8c05b2d9a300 x1874588654899584/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757314.134847:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757314.134860:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757314.134874:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757314.134944:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654900096/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757314.134950:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588654899584/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757314.134988:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757314.134993:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757314.134996:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787757314.135005:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757314.135008:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787757314.135016:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10370 00000100:00100000:0.0:1787757314.135021:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654899584:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787757314.135025:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757314.135038:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757314.135045:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654899712:192.168.201.107@tcp:38:kworker.0 00000008:00100000:1.0:1787757314.135049:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757314.135072:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654900224/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757314.135086:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757314.135091:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757314.135093:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757314.135098:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10370 00000100:00080000:1.0:1787757314.135105:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787757314.135113:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757314.135115:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757314.135129:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757314.135147:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588654900352/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757314.135160:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757314/real 1787757314] req@ffff8c05b2d99f80 x1874588654899712/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757314.135195:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654899712/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757314.135212:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757314.135219:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757314.135227:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654899712:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757314.135241:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654899840:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757314.135284:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757314.135325:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757314/real 1787757314] req@ffff8c05b2d98000 x1874588654899840/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757314.135342:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654899840/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757314.135353:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757314.135357:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757314.135361:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654899840:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757314.135384:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654899968:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757314.135409:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757314.135422:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757314/real 1787757314] req@ffff8c05b2d9b480 x1874588654899968/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757314.135437:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654899968/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757314.135447:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757314.135450:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757314.135454:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654899968:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757314.135465:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654900096:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757314.135497:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757314.135519:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757314/real 1787757314] req@ffff8c05b2d98380 x1874588654900096/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757314.135544:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654900096/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757314.135561:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757314.135568:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757314.135573:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654900096:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757314.135582:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654900224:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757314.135609:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757314.135623:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757314/real 1787757314] req@ffff8c05b2d99880 x1874588654900224/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757314.135638:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654900224/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757314.135648:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757314.135652:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757314.135656:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654900224:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757314.135662:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654900352:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757314.135680:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757314.135691:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757314/real 1787757314] req@ffff8c05b2d9b100 x1874588654900352/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757314.135704:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654900352/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757314.135713:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757314.135717:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757314.135721:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654900352:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757314.261948:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757314.710033:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757315.286004:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757315.734193:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757316.309951:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757316.758166:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757317.334029:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757317.782191:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757318.358007:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757318.806193:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757319.253998:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757319.254009:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787757319.254014:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757319.254024:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10375 00000100:00080000:1.0:1787757319.254036:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757319.254043:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757319.254085:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588654900480/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757319.254117:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757319.254121:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757319.254124:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757319.254128:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10375 00000100:00080000:1.0:1787757319.254131:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757319.254135:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757319.254145:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757319.254161:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588654900608/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757319.254173:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757319.254179:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787757319.254198:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654900480:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757319.254200:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757319.254215:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10375 00000100:00080000:1.0:1787757319.254220:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757319.254225:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757319.254233:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757319.254251:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588654900736/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757319.254266:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757319.254284:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757319.254286:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757319.254313:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10375 00000100:00080000:1.0:1787757319.254317:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757319.254324:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:0.0:1787757319.254333:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:1.0:1787757319.254334:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757319.254352:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654900864/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757319.254363:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757319.254367:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757319.254369:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757319.254372:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10375 00000100:00080000:1.0:1787757319.254376:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757319.254381:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757319.254386:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757319.254397:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757319/real 1787757319] req@ffff8c05b2d9aa00 x1874588654900480/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757319.254401:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654900992/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757319.254451:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757319.254455:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757319.254457:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757319.254464:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588654900480/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757319.254467:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10375 00000100:00080000:1.0:1787757319.254475:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757319.254484:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757319.254494:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787757319.254496:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757319.254505:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757319.254509:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588654901120/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757319.254516:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654900480:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757319.254534:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757319.254538:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787757319.254540:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757319.254545:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10375 00000100:00080000:1.0:1787757319.254548:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757319.254552:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757319.254558:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654900608:192.168.201.107@tcp:38:kworker.0 00000008:00100000:1.0:1787757319.254559:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757319.254599:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654901248/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757319.254613:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757319.254617:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757319.254620:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757319.254624:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10375 00000100:00080000:1.0:1787757319.254641:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757319.254651:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:0.0:1787757319.254656:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:1.0:1787757319.254694:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757319.254716:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654901376/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757319.254749:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757319/real 1787757319] req@ffff8c05b2d9bb80 x1874588654900608/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757319.254777:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588654900608/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757319.254794:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757319.254798:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757319.254802:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654900608:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757319.254810:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654900736:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757319.254837:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757319.254869:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757319/real 1787757319] req@ffff8c05b2d9b100 x1874588654900736/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757319.254884:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654900736/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757319.254894:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757319.254898:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757319.254902:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654900736:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757319.254975:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654900864:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757319.255012:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757319.255032:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757319/real 1787757319] req@ffff8c05b2d99880 x1874588654900864/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757319.255054:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654900864/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757319.255076:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757319.255080:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757319.255084:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654900864:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757319.255091:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654900992:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757319.255116:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757319.255128:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757319/real 1787757319] req@ffff8c05b2d98380 x1874588654900992/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757319.255139:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654900992/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757319.255148:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757319.255151:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757319.255154:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654900992:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757319.255159:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654901120:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757319.255175:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757319.255201:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757319/real 1787757319] req@ffff8c05b2d9b480 x1874588654901120/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757319.255214:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654901120/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757319.255227:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757319.255233:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757319.255238:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654901120:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757319.255274:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654901248:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757319.255328:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757319.255345:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757319/real 1787757319] req@ffff8c05b2d98000 x1874588654901248/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757319.255358:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654901248/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757319.255366:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757319.255369:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757319.255373:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654901248:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757319.255379:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654901376:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757319.255398:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757319.255408:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757319/real 1787757319] req@ffff8c05b2d99f80 x1874588654901376/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757319.255419:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654901376/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757319.255427:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757319.255430:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757319.255433:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654901376:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757319.382009:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757319.830192:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757320.406071:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757320.854220:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757321.429997:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757321.878171:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757322.454031:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757322.902147:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757323.477987:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757323.926119:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757324.374042:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757324.374064:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757324.374073:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757324.374092:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10380 00000100:00080000:3.0:1787757324.374109:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757324.374120:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757324.374184:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588654901504/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757324.374239:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757324.374247:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757324.374251:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757324.374262:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10380 00000100:00080000:3.0:1787757324.374273:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757324.374283:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757324.374302:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654901504:192.168.201.107@tcp:250:kworker.0 00000008:00100000:3.0:1787757324.374302:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757324.374345:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588654901632/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757324.374366:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757324.374371:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757324.374374:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757324.374381:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10380 00000100:00080000:3.0:1787757324.374387:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787757324.374390:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757324.374408:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757324.374422:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757324.374442:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757324/real 1787757324] req@ffff8c05a9f7b800 x1874588654901504/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757324.374456:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588654901760/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757324.374467:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588654901504/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757324.374515:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757324.374516:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757324.374531:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757324.374533:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757324.374539:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654901504:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787757324.374565:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654901632:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757324.374599:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757324.374611:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10380 00000800:00000100:0.0:1787757324.374622:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757324.374630:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757324.374641:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757324/real 1787757324] req@ffff8c05a9f7b100 x1874588654901632/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757324.374642:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757324.374666:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757324.374677:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588654901632/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757324.374691:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757324.374699:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757324.374704:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588654901888/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757324.374714:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654901632:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757324.374740:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787757324.374748:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654901760:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757324.374751:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757324.374759:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757324.374766:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10380 00000100:00080000:3.0:1787757324.374774:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757324.374780:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:0.0:1787757324.374788:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:3.0:1787757324.374796:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757324.374812:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757324/real 1787757324] req@ffff8c05a9f79500 x1874588654901760/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757324.374819:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588654902016/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757324.374853:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757324.374858:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757324.374860:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757324.374866:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588654901760/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757324.374923:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757324.374927:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757324.374930:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654901760:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757324.374935:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10380 00000100:00080000:3.0:1787757324.374943:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757324.374951:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757324.374959:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654901888:192.168.201.107@tcp:8:kworker.0 00000008:00100000:3.0:1787757324.374964:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787757324.374994:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757324.374997:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588654902144/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757324.375008:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757324/real 1787757324] req@ffff8c05a9f78e00 x1874588654901888/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757324.375065:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787757324.375067:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588654901888/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757324.375090:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787757324.375095:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757324.375096:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757324.375107:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787757324.375114:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10380 00000100:00100000:0.0:1787757324.375123:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654901888:192.168.201.107@tcp:8:kworker.0 00000100:00080000:3.0:1787757324.375133:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757324.375142:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654902016:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757324.375150:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757324.375160:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787757324.375169:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757324.375198:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588654902272/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757324.375199:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757324/real 1787757324] req@ffff8c05a9f79f80 x1874588654902016/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757324.375294:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588654902016/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757324.375298:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757324.375309:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757324.375312:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757324.375320:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787757324.375320:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757324.375330:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654902016:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757324.375334:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10380 00000100:00080000:3.0:1787757324.375352:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757324.375362:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757324.375364:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654902144:192.168.201.107@tcp:38:kworker.0 00000008:00100000:3.0:1787757324.375379:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787757324.375393:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757324.375412:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757324/real 1787757324] req@ffff8c05a9f79180 x1874588654902144/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757324.375412:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588654902400/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757324.375481:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588654902144/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757324.375512:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757324.375517:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757324.375522:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654902144:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757324.375529:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654902272:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757324.375544:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757324.375553:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757324/real 1787757324] req@ffff8c05a9f7bb80 x1874588654902272/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757324.375560:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588654902272/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757324.375566:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757324.375581:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757324.375583:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654902272:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757324.375595:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654902400:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757324.375610:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757324.375618:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757324/real 1787757324] req@ffff8c05a9f7ad80 x1874588654902400/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757324.375625:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588654902400/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757324.375631:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757324.375633:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757324.375636:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654902400:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757324.501966:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757324.949999:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757325.525962:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757325.974236:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757326.550003:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757326.998125:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757327.574039:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757328.022251:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757328.598004:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757329.046070:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757329.494004:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757329.494017:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757329.494023:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757329.494035:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10385 00000100:00080000:3.0:1787757329.494048:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757329.494057:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757329.494102:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588654902528/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757329.494151:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757329.494156:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757329.494159:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757329.494165:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10385 00000100:00080000:3.0:1787757329.494170:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757329.494175:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757329.494189:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757329.494199:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654902528:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757329.494221:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588654902656/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757329.494259:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757329.494267:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757329.494271:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757329.494280:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10385 00000100:00080000:3.0:1787757329.494289:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787757329.494301:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757329.494304:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757329.494317:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757329.494338:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588654902784/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757329.494354:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757329.494359:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757329.494364:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757329.494366:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757329/real 1787757329] req@ffff8c05a9f79880 x1874588654902528/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757329.494374:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10385 00000100:00080000:3.0:1787757329.494386:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757329.494398:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757329.494408:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757329.494414:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588654902528/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757329.494428:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588654902912/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757329.494439:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757329.494451:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757329.494457:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787757329.494460:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757329.494465:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757329.494472:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10385 00000100:00080000:3.0:1787757329.494476:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757329.494502:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654902528:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757329.494520:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757329.494533:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757329.494543:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654902656:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757329.494563:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588654903040/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757329.494583:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757329.494597:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757329.494607:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787757329.494611:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757329/real 1787757329] req@ffff8c05a9f7a300 x1874588654902656/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757329.494613:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757329.494629:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10385 00000100:00080000:3.0:1787757329.494644:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757329.494654:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588654902656/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757329.494660:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757329.494682:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787757329.494683:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757329.494695:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757329.494702:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654902656:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757329.494711:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654902784:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757329.494722:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588654903168/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757329.494740:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757329.494762:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757329/real 1787757329] req@ffff8c05a9f7aa00 x1874588654902784/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757329.494777:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787757329.494787:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588654902784/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757329.494789:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757329.494797:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757329.494815:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10385 00000100:00080000:0.0:1787757329.494818:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757329.494823:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757329.494830:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787757329.494839:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757329.494846:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654902784:192.168.201.107@tcp:38:kworker.0 00000008:00100000:3.0:1787757329.494854:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757329.494861:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654902912:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757329.494947:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588654903296/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757329.494964:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787757329.494966:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757329.494976:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757329.494980:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757329.494991:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10385 00000100:00080000:3.0:1787757329.495000:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757329.495001:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757329/real 1787757329] req@ffff8c05a9f78700 x1874588654902912/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757329.495013:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757329.495030:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757329.495040:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588654902912/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757329.495053:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588654903424/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757329.495058:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757329.495074:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757329.495087:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654902912:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757329.495115:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654903040:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757329.495141:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757329.495154:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757329/real 1787757329] req@ffff8c05a9f7b480 x1874588654903040/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757329.495168:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588654903040/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757329.495179:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757329.495183:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757329.495187:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654903040:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757329.495193:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654903168:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757329.495211:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757329.495222:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757329/real 1787757329] req@ffff8c05a9f78a80 x1874588654903168/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757329.495235:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588654903168/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757329.495245:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757329.495249:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757329.495252:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654903168:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757329.495258:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654903296:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757329.495275:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757329.495285:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757329/real 1787757329] req@ffff8c05a9f7a680 x1874588654903296/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757329.495298:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588654903296/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757329.495308:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757329.495312:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757329.495316:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654903296:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757329.495323:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654903424:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757329.495340:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757329.495350:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757329/real 1787757329] req@ffff8c05a9f78380 x1874588654903424/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757329.495363:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588654903424/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757329.495373:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757329.495377:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757329.495380:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654903424:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757329.622047:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757330.070221:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757330.646016:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757331.094143:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757331.669962:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757332.118057:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757332.694001:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757333.142101:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757333.717992:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757334.166198:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757334.613965:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757334.613979:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757334.613985:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757334.613997:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10391 00000100:00080000:3.0:1787757334.614022:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757334.614032:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757334.614076:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588654903552/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757334.614113:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757334.614120:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757334.614124:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757334.614132:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10391 00000100:00080000:3.0:1787757334.614139:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757334.614146:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757334.614161:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757334.614188:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588654903680/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757334.614191:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654903552:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757334.614228:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757334.614235:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757334.614238:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757334.614255:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10391 00000100:00080000:3.0:1787757334.614261:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757334.614268:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757334.614277:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757334.614299:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588654903808/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757334.614343:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757334.614350:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787757334.614353:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654903680:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757334.614354:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757334.614371:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10391 00000100:00080000:3.0:1787757334.614378:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757334.614385:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757334.614415:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757334.614440:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654903808:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757334.614441:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588654903936/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757334.614463:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757334.614470:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757334.614474:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757334.614482:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10391 00000100:00080000:3.0:1787757334.614488:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757334.614492:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654903936:192.168.201.107@tcp:8:kworker.0 00000100:00080000:3.0:1787757334.614499:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757334.614508:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757334.614532:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588654904064/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757334.614564:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757334.614572:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757334.614576:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757334.614584:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10391 00000100:00080000:3.0:1787757334.614593:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757334.614600:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757334.614611:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757334.614614:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654904064:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757334.614635:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588654904192/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757334.614666:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757334.614672:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787757334.614673:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654904192:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757334.614676:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757334.614683:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10391 00000100:00080000:3.0:1787757334.614690:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757334.614696:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757334.614705:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757334.614726:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588654904320/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757334.614756:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757334.614763:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757334.614767:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757334.614773:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10391 00000100:00080000:3.0:1787757334.614781:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757334.614802:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654904320:192.168.201.107@tcp:8:kworker.0 00000100:00080000:3.0:1787757334.614806:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757334.614815:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757334.614836:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588654904448/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757334.614933:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654904448:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757334.742077:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757335.190088:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757335.766067:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757336.214288:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757336.790012:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757337.238106:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787757337.686452:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.107:988 00000400:00000100:1.0:1787757337.686554:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.107@tcp at host 192.168.201.107 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787757337.686595:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787757337.686698:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787757337.686741:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787757337.686760:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787757337.686785:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787757337.686816:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:0.0:1787757337.686819:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757334/real 1787757337] req@ffff8c05a9f7b480 x1874588654903936/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787757337.686839:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787757337.686869:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:0.0:1787757337.686954:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588654903936/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757337.686980:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757337.686995:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757337.687007:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654903936:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757337.687024:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757334/real 1787757337] req@ffff8c05a9f7a680 x1874588654903680/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757337.687056:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588654903680/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757337.687077:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757337.687086:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757337.687095:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654903680:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757337.687112:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757334/real 1787757337] req@ffff8c05a9f78a80 x1874588654903808/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757337.687138:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588654903808/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757337.687154:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757337.687160:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757337.687180:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654903808:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757337.687195:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757334/real 1787757337] req@ffff8c05a9f78380 x1874588654903552/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757337.687222:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588654903552/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757337.687239:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757337.687256:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757337.687264:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654903552:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787757337.687316:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757334/real 1787757337] req@ffff8c05a9f79880 x1874588654904448/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757337.687403:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588654904448/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757337.687420:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757337.687428:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757337.687435:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654904448:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757337.687448:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757334/real 1787757337] req@ffff8c05a9f7a300 x1874588654904320/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757337.687470:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588654904320/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757337.687485:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757337.687491:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757337.687497:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654904320:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757337.687510:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757334/real 1787757337] req@ffff8c05a9f7aa00 x1874588654904192/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757337.687530:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588654904192/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757337.687547:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757337.687555:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757337.687562:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654904192:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757337.687576:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757334/real 1787757337] req@ffff8c05a9f78700 x1874588654904064/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757337.687600:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588654904064/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757337.687621:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757337.687628:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757337.687636:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654904064:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757337.814034:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757338.262238:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757338.838003:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757339.286037:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757339.734068:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757339.734085:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787757339.734100:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757339.734123:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10396 00000100:00080000:1.0:1787757339.734138:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757339.734146:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757339.734190:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588654904576/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757339.734228:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757339.734233:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757339.734236:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757339.734241:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10396 00000100:00080000:1.0:1787757339.734245:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757339.734249:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757339.734260:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757339.734274:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588654904704/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757339.734326:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757339.734332:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757339.734335:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757339.734341:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10396 00000100:00080000:1.0:1787757339.734350:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757339.734358:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757339.734371:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757339.734387:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654904576:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787757339.734407:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654904832/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757339.734438:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757339.734445:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757339.734449:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757339.734458:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10396 00000100:00080000:1.0:1787757339.734466:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757339.734475:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757339.734486:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757339.734512:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654904960/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757339.734519:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757339.734547:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757339.734555:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757339.734560:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757339.734574:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10396 00000100:00080000:1.0:1787757339.734585:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757339.734595:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757339.734609:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757339.734629:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757339/real 1787757339] req@ffff8c05b2d9a300 x1874588654904576/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757339.734641:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588654905088/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757339.734703:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757339.734716:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787757339.734718:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588654904576/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757339.734723:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757339.734749:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10396 00000100:00080000:1.0:1787757339.734764:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757339.734784:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787757339.734806:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787757339.734807:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757339.734843:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654905216/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757339.734849:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757339.734871:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654904576:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787757339.734989:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654904704:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787757339.735011:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757339.735020:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787757339.735025:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757339.735039:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10396 00000800:00000100:0.0:1787757339.735053:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757339.735059:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757339.735076:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757339.735091:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757339.735093:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757339/real 1787757339] req@ffff8c05b2d98700 x1874588654904704/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757339.735135:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654904704/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757339.735139:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654905344/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757339.735197:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757339.735219:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757339.735238:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654904704:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787757339.735253:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757339.735263:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757339.735267:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757339.735278:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10396 00000100:00080000:1.0:1787757339.735286:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757339.735292:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757339.735328:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757339.735336:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654904832:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757339.735363:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588654905472/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757339.735392:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757339.735422:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757339/real 1787757339] req@ffff8c05b2d99f80 x1874588654904832/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757339.735447:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654904832/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757339.735461:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757339.735468:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757339.735474:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654904832:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757339.735487:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654904960:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757339.735526:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757339.735557:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757339/real 1787757339] req@ffff8c05b2d98000 x1874588654904960/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757339.735576:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654904960/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757339.735589:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757339.735595:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757339.735600:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654904960:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757339.735611:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654905088:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757339.735646:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757339.735663:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757339/real 1787757339] req@ffff8c05b2d9b480 x1874588654905088/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757339.735680:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654905088/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757339.735692:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757339.735698:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757339.735703:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654905088:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757339.735713:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654905216:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757339.735744:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757339.735763:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757339/real 1787757339] req@ffff8c05b2d98380 x1874588654905216/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757339.735782:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654905216/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757339.735796:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757339.735801:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757339.735814:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654905216:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757339.735825:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654905344:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757339.735858:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757339.735918:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757339/real 1787757339] req@ffff8c05b2d99880 x1874588654905344/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757339.735940:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654905344/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757339.735954:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757339.735959:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757339.735965:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654905344:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757339.736013:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654905472:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757339.736046:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757339.736064:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757339/real 1787757339] req@ffff8c05b2d9b100 x1874588654905472/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757339.736080:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654905472/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757339.736100:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757339.736105:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757339.736110:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654905472:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757339.862052:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757340.310262:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757340.885977:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757341.334157:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757341.909998:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757342.358139:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757342.934042:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757343.383621:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757343.958021:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757344.406284:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757344.854051:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757344.854067:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787757344.854074:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757344.854092:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10401 00000100:00080000:1.0:1787757344.854109:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757344.854120:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757344.854194:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588654905600/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757344.854288:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757344.854296:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757344.854301:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757344.854312:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10401 00000100:00080000:1.0:1787757344.854322:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757344.854333:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757344.854353:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757344.854373:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654905600:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787757344.854387:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588654905728/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757344.854411:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757344.854417:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757344.854420:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757344.854427:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10401 00000100:00080000:1.0:1787757344.854433:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757344.854439:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757344.854446:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757344.854467:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588654905856/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757344.854494:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757344.854500:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757344.854507:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787757344.854508:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757344.854520:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10401 00000100:00080000:1.0:1787757344.854528:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757344.854534:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757344.854545:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757344.854566:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654905984/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757344.854581:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757344.854587:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1787757344.854595:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757344/real 1787757344] req@ffff8c05b2d9bb80 x1874588654905600/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757344.854596:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757344.854612:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10401 00000100:00080000:1.0:1787757344.854624:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757344.854635:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757344.854643:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757344.854651:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588654905600/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757344.854671:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757344.854678:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757344.854686:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654906112/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757344.854694:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654905600:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757344.854721:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757344.854738:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757344.854741:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757344.854750:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10401 00000100:00080000:1.0:1787757344.854755:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757344.854760:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757344.854766:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757344.854781:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588654906240/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757344.854793:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757344.854798:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787757344.854801:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757344.854806:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10401 00000100:00080000:1.0:1787757344.854810:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757344.854817:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757344.854823:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757344.854834:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654906368/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757344.854847:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757344.854851:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757344.854854:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757344.854858:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10401 00000100:00080000:1.0:1787757344.854862:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757344.854868:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757344.854912:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757344.854929:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654906496/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757344.854959:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654905728:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757344.855052:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757344.855090:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757344/real 1787757344] req@ffff8c05b2d9aa00 x1874588654905728/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757344.855114:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588654905728/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757344.855128:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757344.855134:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757344.855143:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654905728:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757344.855155:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654905856:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757344.855192:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757344.855234:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757344/real 1787757344] req@ffff8c05b2d9b100 x1874588654905856/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757344.855260:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654905856/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757344.855270:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757344.855277:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757344.855284:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654905856:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757344.855303:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654905984:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757344.855342:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757344.855367:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757344/real 1787757344] req@ffff8c05b2d99880 x1874588654905984/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757344.855383:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654905984/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757344.855390:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757344.855393:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757344.855395:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654905984:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757344.855400:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654906112:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757344.855414:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757344.855422:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757344/real 1787757344] req@ffff8c05b2d98380 x1874588654906112/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757344.855432:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654906112/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757344.855438:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757344.855440:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757344.855442:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654906112:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757344.855447:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654906240:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757344.855460:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757344.855467:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757344/real 1787757344] req@ffff8c05b2d9b480 x1874588654906240/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757344.855476:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654906240/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757344.855482:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757344.855484:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757344.855486:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654906240:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757344.855491:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654906368:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757344.855502:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757344.855509:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757344/real 1787757344] req@ffff8c05b2d98000 x1874588654906368/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757344.855523:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654906368/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757344.855530:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757344.855532:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757344.855534:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654906368:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757344.855562:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654906496:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757344.855575:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757344.855583:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757344/real 1787757344] req@ffff8c05b2d99f80 x1874588654906496/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757344.855592:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654906496/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757344.855598:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757344.855600:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757344.855603:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654906496:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757344.982083:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757345.430290:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757346.005986:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757346.454304:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757347.030062:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757347.478185:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757348.054020:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757348.502278:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757349.078021:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757349.526177:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757349.974009:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757349.974021:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787757349.974025:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757349.974035:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10406 00000100:00080000:1.0:1787757349.974047:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757349.974054:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757349.974100:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588654906624/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757349.974143:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757349.974147:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757349.974148:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757349.974152:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10406 00000100:00080000:1.0:1787757349.974155:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757349.974159:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757349.974169:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757349.974180:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588654906752/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757349.974185:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654906624:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757349.974191:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757349.974195:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757349.974197:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757349.974200:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10406 00000100:00080000:1.0:1787757349.974202:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757349.974205:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757349.974209:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757349.974218:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654906880/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757349.974226:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757349.974229:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757349.974230:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757349.974232:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10406 00000100:00080000:1.0:1787757349.974252:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757349.974255:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757349.974259:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787757349.974264:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757349.974269:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654907008/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757349.974277:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757349.974280:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757349.974282:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757349.974285:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10406 00000100:00080000:1.0:1787757349.974292:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757349.974295:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757349.974298:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757349.974305:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588654907136/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757349.974311:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757349.974314:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757349.974316:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757349.974319:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757349/real 1787757349] req@ffff8c05b2d98700 x1874588654906624/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757349.974322:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10406 00000100:00080000:1.0:1787757349.974327:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757349.974334:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757349.974339:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757349.974347:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654906624/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757349.974348:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654907264/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757349.974370:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757349.974380:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787757349.974380:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757349.974385:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787757349.974388:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654906624:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757349.974388:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757349.974395:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10406 00000100:00080000:1.0:1787757349.974398:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757349.974401:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757349.974403:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757349.974410:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654907392/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757349.974417:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654906752:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787757349.974417:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757349.974425:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757349.974427:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757349.974431:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10406 00000100:00080000:1.0:1787757349.974433:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757349.974436:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757349.974439:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787757349.974444:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757349.974449:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588654907520/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757349.974458:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757349/real 1787757349] req@ffff8c05b2d9a300 x1874588654906752/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757349.974468:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588654906752/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757349.974476:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757349.974479:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757349.974481:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654906752:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757349.974487:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654906880:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757349.974500:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757349.974506:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757349/real 1787757349] req@ffff8c05b2d99f80 x1874588654906880/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757349.974515:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654906880/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757349.974525:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757349.974527:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757349.974530:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654906880:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757349.974533:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654907008:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757349.974545:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757349.974550:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757349/real 1787757349] req@ffff8c05b2d98000 x1874588654907008/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757349.974558:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654907008/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757349.974563:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757349.974565:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757349.974567:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654907008:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757349.974571:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654907136:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757349.974581:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757349.974587:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757349/real 1787757349] req@ffff8c05b2d9b480 x1874588654907136/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757349.974594:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654907136/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757349.974602:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757349.974605:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757349.974609:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654907136:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757349.974615:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654907264:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757349.974637:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757349.974649:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757349/real 1787757349] req@ffff8c05b2d98380 x1874588654907264/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757349.974661:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654907264/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757349.974667:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757349.974669:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757349.974672:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654907264:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757349.974686:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654907392:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757349.974701:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757349.974708:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757349/real 1787757349] req@ffff8c05b2d99880 x1874588654907392/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757349.974716:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654907392/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757349.974722:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757349.974724:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757349.974726:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654907392:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757349.974729:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654907520:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757349.974739:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757349.974745:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757349/real 1787757349] req@ffff8c05b2d9b100 x1874588654907520/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757349.974753:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654907520/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757349.974759:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757349.974760:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757349.974762:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654907520:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757350.101962:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757350.550225:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757351.126085:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757351.574239:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757352.150052:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757352.598288:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757353.174015:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757353.622171:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757354.198041:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757354.646239:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757355.093994:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757355.094006:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787757355.094011:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757355.094054:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10411 00000100:00080000:1.0:1787757355.094086:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757355.094095:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757355.094143:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588654907648/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757355.094178:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757355.094183:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757355.094185:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757355.094190:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10411 00000100:00080000:1.0:1787757355.094194:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757355.094198:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757355.094209:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757355.094225:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588654907776/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757355.094238:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757355.094243:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757355.094245:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757355.094249:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10411 00000100:00100000:0.0:1787757355.094266:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654907648:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757355.094266:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757355.094278:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757355.094317:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757355.094334:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588654907904/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757355.094354:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757355.094358:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757355.094360:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757355.094365:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10411 00000100:00080000:1.0:1787757355.094384:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757355.094389:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757355.094394:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787757355.094403:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757355.094410:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654908032/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757355.094422:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757355.094426:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757355.094429:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757355.094433:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10411 00000100:00080000:1.0:1787757355.094439:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757355.094442:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757355.094447:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757355.094458:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654908160/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757355.094467:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757355.094471:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757355.094472:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757355.094478:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10411 00000100:00080000:1.0:1787757355.094483:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757355.094491:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757355/real 1787757355] req@ffff8c05b2d9aa00 x1874588654907648/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757355.094492:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757355.094511:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757355.094530:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588654908288/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757355.094539:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588654907648/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757355.094547:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757355.094561:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787757355.094568:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757355.094573:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757355.094577:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10411 00000100:00080000:0.0:1787757355.094603:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787757355.094605:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757355.094615:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654907648:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757355.094618:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757355.094633:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757355.094640:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654907776:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757355.094651:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654908416/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757355.094668:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757355.094674:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757355.094678:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757355.094685:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10411 00000100:00080000:1.0:1787757355.094694:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787757355.094702:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757355.094710:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757355.094746:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757355.094752:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757355/real 1787757355] req@ffff8c05b2d9bb80 x1874588654907776/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757355.094778:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588654907776/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757355.094780:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654908544/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757355.094803:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757355.094811:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757355.094819:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654907776:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757355.094858:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654907904:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757355.094883:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757355.094926:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757355/real 1787757355] req@ffff8c05b2d9b100 x1874588654907904/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757355.094941:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654907904/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757355.094949:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757355.094953:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757355.094956:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654907904:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757355.094962:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654908032:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757355.094983:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757355.094994:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757355/real 1787757355] req@ffff8c05b2d99880 x1874588654908032/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757355.095013:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654908032/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757355.095021:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757355.095023:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757355.095047:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654908032:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757355.095058:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654908160:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757355.095088:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757355.095100:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757355/real 1787757355] req@ffff8c05b2d98380 x1874588654908160/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757355.095112:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654908160/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757355.095120:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757355.095123:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757355.095126:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654908160:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757355.095132:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654908288:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757355.095147:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757355.095156:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757355/real 1787757355] req@ffff8c05b2d9b480 x1874588654908288/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757355.095167:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654908288/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757355.095181:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757355.095184:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757355.095187:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654908288:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757355.095192:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654908416:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757355.095207:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757355.095216:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757355/real 1787757355] req@ffff8c05b2d98000 x1874588654908416/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757355.095227:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654908416/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757355.095235:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757355.095237:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757355.095240:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654908416:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757355.095246:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654908544:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757355.095260:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757355.095268:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757355/real 1787757355] req@ffff8c05b2d99f80 x1874588654908544/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757355.095303:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654908544/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757355.095317:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757355.095323:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757355.095331:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654908544:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757355.222085:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757355.670060:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757356.246117:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757356.694253:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757357.270064:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757357.718253:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757358.294079:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757358.742220:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757359.318100:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757359.766137:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757360.214093:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757360.214109:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787757360.214117:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757360.214134:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10416 00000100:00080000:2.0:1787757360.214150:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757360.214163:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757360.214224:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588654908672/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757360.214274:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757360.214292:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757360.214296:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757360.214305:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10416 00000100:00080000:2.0:1787757360.214314:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757360.214325:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757360.214342:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757360.214372:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588654908800/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757360.214377:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654908672:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757360.214414:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757360.214423:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757360.214427:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757360.214438:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10416 00000100:00080000:2.0:1787757360.214449:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757360.214458:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757360.214471:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787757360.214480:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757360.214506:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588654908928/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757360.214532:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757360.214542:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787757360.214545:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757360/real 1787757360] req@ffff8c05b2e73100 x1874588654908672/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757360.214548:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757360.214567:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10416 00000100:00080000:2.0:1787757360.214580:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757360.214586:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588654908672/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757360.214597:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787757360.214613:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787757360.214616:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787757360.214624:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757360.214633:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654908672:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787757360.214652:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588654909056/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757360.214668:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654908800:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757360.214692:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757360.214702:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757360.214706:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787757360.214712:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757360.214722:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10416 00000100:00080000:2.0:1787757360.214734:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757360.214737:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757360/real 1787757360] req@ffff8c05b2e72d80 x1874588654908800/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757360.214749:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757360.214766:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757360.214769:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588654908800/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757360.214785:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757360.214794:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757360.214795:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588654909184/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757360.214810:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654908800:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757360.214827:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787757360.214835:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654908928:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757360.214837:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757360.214846:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757360.214856:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10416 00000100:00080000:2.0:1787757360.214863:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757360.214870:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:0.0:1787757360.214964:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:2.0:1787757360.214972:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757360.214988:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757360/real 1787757360] req@ffff8c05b2e72680 x1874588654908928/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757360.215007:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588654909312/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757360.215015:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588654908928/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757360.215060:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757360.215065:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757360.215067:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757360.215080:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654908928:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757360.215082:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787757360.215093:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757360.215105:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10416 00000100:00080000:2.0:1787757360.215113:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757360.215115:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654909056:192.168.201.107@tcp:8:kworker.0 00000100:00080000:2.0:1787757360.215126:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757360.215138:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787757360.215159:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757360.215166:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588654909440/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757360.215184:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757360/real 1787757360] req@ffff8c05b2e70e00 x1874588654909056/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757360.215186:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757360.215200:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757360.215208:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757360.215222:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10416 00000100:00080000:2.0:1787757360.215229:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757360.215231:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588654909056/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757360.215241:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757360.215254:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787757360.215260:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757360.215269:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757360.215276:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588654909568/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757360.215283:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654909056:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757360.215309:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654909184:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757360.215339:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757360.215355:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757360/real 1787757360] req@ffff8c05b2e70000 x1874588654909184/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757360.215369:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588654909184/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757360.215379:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757360.215383:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757360.215386:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654909184:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757360.215392:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654909312:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757360.215411:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757360.215422:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757360/real 1787757360] req@ffff8c05b2e72300 x1874588654909312/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757360.215436:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588654909312/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757360.215445:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757360.215449:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757360.215452:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654909312:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757360.215468:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654909440:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757360.215487:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757360.215498:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757360/real 1787757360] req@ffff8c05b2e72a00 x1874588654909440/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757360.215512:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588654909440/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757360.215522:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757360.215526:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757360.215530:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654909440:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757360.215536:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654909568:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757360.215554:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757360.215565:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757360/real 1787757360] req@ffff8c05b2e71180 x1874588654909568/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757360.215578:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588654909568/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757360.215588:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757360.215591:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757360.215595:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654909568:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757360.342066:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757360.790228:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757361.366073:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757361.814247:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757362.390131:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757362.838309:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757363.414046:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757363.862249:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757364.438046:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757364.886079:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757365.333982:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757365.333995:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757365.334000:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757365.334012:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10421 00000100:00080000:3.0:1787757365.334025:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757365.334033:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757365.334077:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588654909696/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757365.334108:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757365.334112:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757365.334114:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757365.334119:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10421 00000100:00080000:3.0:1787757365.334124:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757365.334129:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757365.334140:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757365.334153:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654909696:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757365.334156:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588654909824/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757365.334171:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757365.334175:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757365.334178:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757365.334182:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10421 00000100:00080000:3.0:1787757365.334185:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757365.334189:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757365.334194:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757365.334205:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588654909952/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757365.334216:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757365.334221:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757365.334223:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757365.334227:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10421 00000800:00000100:0.0:1787757365.334232:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757365.334234:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757365.334241:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757365.334247:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757365.334259:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588654910080/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757365.334276:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787757365.334281:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757365/real 1787757365] req@ffff8c05a9f7ad80 x1874588654909696/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757365.334282:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757365.334287:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757365.334297:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10421 00000100:00080000:3.0:1787757365.334305:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757365.334311:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757365.334317:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588654909696/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787757365.334318:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787757365.334332:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787757365.334337:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588654910208/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757365.334338:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757365.334344:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654909696:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757365.334358:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757365.334362:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757365.334364:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757365.334369:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10421 00000100:00100000:0.0:1787757365.334374:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654909824:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757365.334374:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757365.334382:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757365.334388:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787757365.334398:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757365.334403:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588654910336/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757365.334413:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757365/real 1787757365] req@ffff8c05a9f7bb80 x1874588654909824/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757365.334419:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757365.334426:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787757365.334428:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588654909824/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757365.334431:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757365.334442:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10421 00000100:00080000:0.0:1787757365.334443:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757365.334450:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787757365.334451:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757365.334453:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654909824:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757365.334459:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757365.334461:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654909952:192.168.201.107@tcp:38:kworker.0 00000008:00100000:3.0:1787757365.334467:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787757365.334484:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757365.334486:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588654910464/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757365.334495:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757365/real 1787757365] req@ffff8c05a9f79180 x1874588654909952/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757365.334500:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757365.334510:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757365.334514:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757365.334515:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588654909952/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757365.334523:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10421 00000100:00080000:3.0:1787757365.334529:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757365.334537:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787757365.334539:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757365.334544:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000008:00100000:3.0:1787757365.334544:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757365.334547:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654909952:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757365.334554:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654910080:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757365.334560:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588654910592/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757365.334571:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757365.334582:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757365/real 1787757365] req@ffff8c05a9f79f80 x1874588654910080/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757365.334591:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588654910080/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757365.334598:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757365.334601:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757365.334603:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654910080:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757365.334606:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654910208:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757365.334617:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757365.334623:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757365/real 1787757365] req@ffff8c05a9f78e00 x1874588654910208/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757365.334630:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588654910208/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757365.334635:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757365.334637:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757365.334640:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654910208:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757365.334650:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654910336:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757365.334662:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757365.334668:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757365/real 1787757365] req@ffff8c05a9f79500 x1874588654910336/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757365.334675:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588654910336/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757365.334681:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757365.334682:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757365.334684:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654910336:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757365.334688:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654910464:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757365.334698:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757365.334704:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757365/real 1787757365] req@ffff8c05a9f7b100 x1874588654910464/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757365.334711:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588654910464/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757365.334717:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757365.334718:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757365.334720:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654910464:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757365.334723:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654910592:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757365.334733:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757365.334740:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757365/real 1787757365] req@ffff8c05a9f7b800 x1874588654910592/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757365.334747:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588654910592/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757365.334753:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757365.334754:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757365.334756:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654910592:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757365.461978:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757365.910039:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757366.485996:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757366.934233:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757367.510014:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757367.958150:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757368.534022:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757368.982246:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757369.558004:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757370.006073:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757370.454025:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757370.454039:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787757370.454045:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757370.454057:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10426 00000100:00080000:2.0:1787757370.454071:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757370.454080:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757370.454137:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588654910720/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757370.454189:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757370.454197:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757370.454211:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757370.454220:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10426 00000100:00080000:2.0:1787757370.454229:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757370.454252:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757370.454266:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757370.454290:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654910720:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787757370.454295:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588654910848/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757370.454324:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757370.454329:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757370.454332:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757370.454338:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10426 00000100:00080000:2.0:1787757370.454343:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757370.454349:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757370.454356:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757370.454370:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588654910976/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757370.454383:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757370.454388:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757370.454391:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787757370.454401:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757370.454402:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10426 00000100:00080000:2.0:1787757370.454414:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757370.454419:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757370.454426:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757370.454442:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588654911104/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757370.454454:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757370.454459:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757370.454461:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757370.454466:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10426 00000100:00080000:2.0:1787757370.454471:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757370.454477:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757370.454483:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757370.454490:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757370/real 1787757370] req@ffff8c05b2e70a80 x1874588654910720/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757370.454503:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588654911232/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757370.454523:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757370.454528:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757370.454535:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757370.454536:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588654910720/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757370.454547:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10426 00000100:00080000:2.0:1787757370.454559:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757370.454571:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787757370.454576:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787757370.454577:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787757370.454594:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757370.454602:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588654911360/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757370.454608:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654910720:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757370.454634:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757370.454639:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787757370.454642:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757370.454648:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10426 00000100:00080000:2.0:1787757370.454652:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757370.454662:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654910848:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757370.454663:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757370.454678:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757370.454699:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588654911488/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757370.454714:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757370.454718:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757370.454725:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787757370.454726:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757370.454737:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10426 00000100:00080000:2.0:1787757370.454745:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757370.454751:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757370.454789:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757370/real 1787757370] req@ffff8c05b2e71180 x1874588654910848/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787757370.454793:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757370.454824:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588654911616/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757370.454839:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588654910848/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757370.454861:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757370.454870:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757370.454922:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654910848:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757370.454933:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654910976:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757370.454969:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757370.454988:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757370/real 1787757370] req@ffff8c05b2e72a00 x1874588654910976/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757370.455004:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588654910976/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757370.455014:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757370.455019:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757370.455023:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654910976:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757370.455031:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654911104:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757370.455054:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757370.455073:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757370/real 1787757370] req@ffff8c05b2e72300 x1874588654911104/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757370.455097:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588654911104/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757370.455115:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757370.455122:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757370.455130:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654911104:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757370.455142:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654911232:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757370.455184:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757370.455209:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757370/real 1787757370] req@ffff8c05b2e70000 x1874588654911232/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757370.455235:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588654911232/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757370.455252:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757370.455259:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757370.455265:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654911232:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757370.455274:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654911360:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757370.455304:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757370.455319:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757370/real 1787757370] req@ffff8c05b2e70e00 x1874588654911360/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757370.455334:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588654911360/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757370.455344:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757370.455348:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757370.455354:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654911360:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757370.455385:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654911488:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757370.455421:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757370.455444:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757370/real 1787757370] req@ffff8c05b2e72680 x1874588654911488/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757370.455470:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588654911488/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757370.455488:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757370.455493:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757370.455497:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654911488:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757370.455506:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654911616:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757370.455534:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757370.455549:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757370/real 1787757370] req@ffff8c05b2e72d80 x1874588654911616/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757370.455563:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588654911616/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757370.455573:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757370.455577:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757370.455581:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654911616:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757370.582030:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757371.030296:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757371.606043:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757372.054141:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757372.629984:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757373.078680:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757373.654023:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757374.102154:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757374.678019:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757375.126335:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757375.574117:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757375.574131:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787757375.574138:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757375.574154:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10431 00000100:00080000:1.0:1787757375.574172:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757375.574184:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757375.574260:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588654911744/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757375.574296:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757375.574301:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757375.574304:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757375.574309:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10431 00000100:00080000:1.0:1787757375.574315:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757375.574320:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757375.574333:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757375.574354:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588654911872/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757375.574382:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654911744:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757375.574383:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757375.574397:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757375.574404:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757375.574411:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10431 00000100:00080000:1.0:1787757375.574415:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757375.574419:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757375.574426:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757375.574443:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654912000/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757375.574457:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757375.574462:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757375.574464:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757375.574468:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10431 00000100:00080000:1.0:1787757375.574484:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757375.574489:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:0.0:1787757375.574499:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:1.0:1787757375.574499:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757375.574522:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654912128/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757375.574535:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757375.574539:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757375.574542:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757375.574547:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10431 00000100:00080000:1.0:1787757375.574555:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757375.574560:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757375.574568:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757375/real 1787757375] req@ffff8c05b2d9a300 x1874588654911744/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787757375.574571:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757375.574597:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588654912256/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757375.574610:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588654911744/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757375.574614:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757375.574629:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757375.574636:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757375.574649:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10431 00000100:00080000:0.0:1787757375.574654:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757375.574655:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757375.574698:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787757375.574706:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000008:00100000:1.0:1787757375.574708:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757375.574719:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654911744:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787757375.574731:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654912384/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757375.574746:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757375.574750:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787757375.574752:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757375.574789:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10431 00000100:00080000:1.0:1787757375.574796:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757375.574801:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654911872:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787757375.574808:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757375.574821:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757375.574841:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654912512/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757375.574864:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757375.574917:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757375.574924:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757375.574928:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757375.574935:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10431 00000100:00080000:1.0:1787757375.574970:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757375.574976:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757375.574985:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757375.574997:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757375/real 1787757375] req@ffff8c05b2d98700 x1874588654911872/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757375.575007:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588654912640/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757375.575079:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654911872/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757375.575098:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757375.575104:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757375.575111:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654911872:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757375.575124:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654912000:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757375.575158:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757375.575179:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757375/real 1787757375] req@ffff8c05b2d99f80 x1874588654912000/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757375.575203:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654912000/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757375.575219:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757375.575226:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757375.575233:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654912000:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757375.575245:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654912128:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757375.575285:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757375.575303:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757375/real 1787757375] req@ffff8c05b2d98000 x1874588654912128/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757375.575319:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654912128/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757375.575331:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757375.575337:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757375.575344:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654912128:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757375.575355:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654912256:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757375.575394:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757375.575419:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757375/real 1787757375] req@ffff8c05b2d9b480 x1874588654912256/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757375.575446:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654912256/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757375.575458:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757375.575461:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757375.575466:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654912256:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757375.575474:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654912384:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757375.575503:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757375.575518:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757375/real 1787757375] req@ffff8c05b2d98380 x1874588654912384/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757375.575532:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654912384/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757375.575542:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757375.575545:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757375.575549:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654912384:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757375.575576:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654912512:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757375.575597:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757375.575609:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757375/real 1787757375] req@ffff8c05b2d99880 x1874588654912512/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757375.575623:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654912512/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757375.575642:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757375.575646:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757375.575650:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654912512:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757375.575656:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654912640:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757375.575716:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757375.575733:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757375/real 1787757375] req@ffff8c05b2d9b100 x1874588654912640/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757375.575747:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654912640/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757375.575758:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757375.575762:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757375.575766:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654912640:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757375.701989:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757376.150116:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757376.726065:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757377.174272:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757377.749978:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757378.198094:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757378.774044:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757379.222279:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757379.798016:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757380.246120:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757380.694004:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757380.694016:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787757380.694022:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757380.694034:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10437 00000100:00080000:2.0:1787757380.694047:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757380.694056:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757380.694120:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588654912768/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757380.694154:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757380.694159:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757380.694162:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757380.694167:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10437 00000100:00080000:2.0:1787757380.694171:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757380.694175:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757380.694186:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757380.694203:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588654912896/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757380.694218:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757380.694223:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757380.694226:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757380.694234:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10437 00000100:00100000:0.0:1787757380.694238:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654912768:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757380.694240:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757380.694252:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757380.694264:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757380.694279:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588654913024/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757380.694292:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757380.694296:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757380.694299:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757380.694304:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10437 00000100:00080000:2.0:1787757380.694307:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757380.694312:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757380.694317:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757380.694329:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588654913152/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757380.694340:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757380.694344:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757380.694346:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757380.694349:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10437 00000100:00080000:2.0:1787757380.694355:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757380.694360:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757380.694364:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757380.694375:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588654913280/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757380.694394:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757380.694407:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757380.694412:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757380.694415:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757380.694420:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10437 00000100:00080000:2.0:1787757380.694424:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757380.694427:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757380.694433:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757380.694448:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588654913408/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757380.694459:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757380.694464:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787757380.694466:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757380.694472:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10437 00000100:00080000:2.0:1787757380.694477:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757380.694482:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757380.694488:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757380.694492:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757380/real 1787757380] req@ffff8c05b2e73100 x1874588654912768/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757380.694507:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588654913536/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757380.694557:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757380.694564:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757380.694567:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757380.694599:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10437 00000100:00100000:0.0:1787757380.694605:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588654912768/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757380.694607:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757380.694621:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757380.694636:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787757380.694640:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757380.694650:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757380.694660:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588654913664/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757380.694662:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654912768:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787757380.694711:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654912896:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757380.694752:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757380.694771:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757380/real 1787757380] req@ffff8c05b2e70700 x1874588654912896/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757380.694788:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588654912896/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757380.694799:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757380.694805:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757380.694811:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654912896:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757380.694820:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654913024:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757380.694850:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757380.694863:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757380/real 1787757380] req@ffff8c05b2e73480 x1874588654913024/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757380.694932:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588654913024/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757380.694952:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757380.694957:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757380.694962:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654913024:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757380.694974:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654913152:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757380.695010:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757380.695030:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757380/real 1787757380] req@ffff8c05b2e71500 x1874588654913152/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757380.695050:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588654913152/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757380.695064:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757380.695069:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757380.695076:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654913152:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757380.695087:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654913280:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757380.695119:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757380.695134:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757380/real 1787757380] req@ffff8c05b2e73b80 x1874588654913280/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757380.695151:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588654913280/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757380.695162:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757380.695169:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757380.695174:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654913280:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757380.695185:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654913408:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757380.695208:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757380.695229:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757380/real 1787757380] req@ffff8c05b2e71c00 x1874588654913408/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757380.695255:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588654913408/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757380.695272:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757380.695279:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757380.695289:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654913408:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757380.695307:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654913536:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757380.695357:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757380.695385:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757380/real 1787757380] req@ffff8c05b2e71880 x1874588654913536/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757380.695403:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588654913536/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757380.695416:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757380.695422:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757380.695429:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654913536:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757380.695441:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654913664:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757380.695470:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757380.695487:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757380/real 1787757380] req@ffff8c05b2e70380 x1874588654913664/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757380.695505:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588654913664/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757380.695516:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757380.695521:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757380.695525:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654913664:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757380.822022:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757381.270260:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757381.846038:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757382.294105:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757382.870001:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757383.318178:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757383.894014:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757384.342173:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757384.918037:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757385.366199:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757385.814067:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757385.814085:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757385.814092:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757385.814104:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10442 00000100:00080000:3.0:1787757385.814118:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757385.814127:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757385.814175:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588654913792/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757385.814216:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757385.814221:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757385.814224:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757385.814229:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10442 00000100:00080000:3.0:1787757385.814233:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757385.814238:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757385.814250:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757385.814267:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588654913920/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757385.814284:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787757385.814287:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654913792:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757385.814290:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757385.814296:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757385.814310:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10442 00000100:00080000:3.0:1787757385.814315:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757385.814319:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757385.814325:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757385.814343:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588654914048/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757385.814356:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757385.814362:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757385.814364:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757385.814369:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10442 00000100:00080000:3.0:1787757385.814374:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757385.814378:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757385.814384:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787757385.814404:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757385.814404:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588654914176/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757385.814423:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757385.814428:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757385.814431:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757385.814437:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10442 00000100:00080000:3.0:1787757385.814441:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757385.814446:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757385.814452:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757385.814470:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757385/real 1787757385] req@ffff8c05a9f7b800 x1874588654913792/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757385.814472:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588654914304/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757385.814539:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787757385.814543:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588654913792/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757385.814546:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757385.814553:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757385.814570:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10442 00000100:00080000:0.0:1787757385.814578:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757385.814579:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757385.814594:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787757385.814597:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000008:00100000:3.0:1787757385.814606:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757385.814614:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654913792:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757385.814629:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588654914432/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757385.814643:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757385.814647:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757385.814650:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757385.814655:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10442 00000100:00080000:3.0:1787757385.814659:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757385.814665:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654913920:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757385.814671:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757385.814689:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757385.814717:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588654914560/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757385.814733:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757385.814754:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787757385.814763:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757385/real 1787757385] req@ffff8c05a9f7b100 x1874588654913920/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757385.814766:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757385.814774:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757385.814796:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10442 00000100:00100000:0.0:1787757385.814802:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588654913920/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757385.814809:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757385.814819:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757385.814829:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787757385.814833:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757385.814846:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654913920:192.168.201.107@tcp:38:kworker.0 00000008:00100000:3.0:1787757385.814852:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757385.814859:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654914048:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757385.814929:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757385.814951:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757385/real 1787757385] req@ffff8c05a9f79500 x1874588654914048/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757385.814958:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588654914688/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757385.814993:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588654914048/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757385.815027:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757385.815036:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757385.815044:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654914048:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757385.815058:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654914176:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757385.815094:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757385.815111:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757385/real 1787757385] req@ffff8c05a9f78e00 x1874588654914176/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757385.815126:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588654914176/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757385.815137:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757385.815141:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757385.815145:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654914176:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757385.815153:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654914304:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757385.815172:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757385.815184:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757385/real 1787757385] req@ffff8c05a9f79f80 x1874588654914304/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757385.815198:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588654914304/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757385.815208:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757385.815211:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757385.815215:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654914304:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757385.815222:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654914432:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757385.815240:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757385.815252:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757385/real 1787757385] req@ffff8c05a9f79180 x1874588654914432/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757385.815274:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588654914432/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757385.815289:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757385.815295:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757385.815303:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654914432:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757385.815342:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654914560:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757385.815377:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757385.815393:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757385/real 1787757385] req@ffff8c05a9f7bb80 x1874588654914560/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757385.815408:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588654914560/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757385.815422:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757385.815429:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757385.815435:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654914560:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757385.815447:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654914688:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757385.815512:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757385.815538:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757385/real 1787757385] req@ffff8c05a9f7ad80 x1874588654914688/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757385.815554:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588654914688/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757385.815564:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757385.815568:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757385.815573:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654914688:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757385.942025:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757386.390124:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757386.966024:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757387.414281:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757387.990064:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757388.438226:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757389.013991:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757389.462239:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757390.038070:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757390.486140:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757390.934125:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757390.934143:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787757390.934150:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757390.934167:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10447 00000100:00080000:2.0:1787757390.934184:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757390.934197:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757390.934259:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73800 x1874588654914816/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757390.934305:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757390.934310:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757390.934312:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757390.934317:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10447 00000100:00080000:2.0:1787757390.934324:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757390.934330:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757390.934342:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757390.934364:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588654914944/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757390.934380:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757390.934390:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787757390.934416:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654914816:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757390.934422:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757390.934442:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10447 00000100:00080000:2.0:1787757390.934449:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757390.934455:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757390.934474:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757390.934495:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588654915072/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757390.934509:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757390.934514:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757390.934517:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757390.934523:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10447 00000100:00080000:2.0:1787757390.934527:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757390.934531:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757390.934538:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757390.934553:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588654915200/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757390.934569:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787757390.934584:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757390.934587:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757390.934592:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757390.934600:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10447 00000100:00080000:2.0:1787757390.934605:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757390.934613:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757390.934624:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757390.934642:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588654915328/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757390.934656:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757390.934661:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757390.934664:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757390.934677:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10447 00000100:00100000:0.0:1787757390.934678:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757390/real 1787757390] req@ffff8c05b2e73800 x1874588654914816/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757390.934685:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757390.934703:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757390.934723:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757390.934764:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588654915456/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757390.934774:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73800 x1874588654914816/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757390.934829:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757390.934839:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787757390.934852:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787757390.934854:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654914816:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757390.934970:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787757390.934977:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757390.934983:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654914944:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757390.934993:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10447 00000100:00080000:2.0:1787757390.935007:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757390.935014:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757390.935023:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787757390.935044:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757390.935052:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588654915584/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757390.935072:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757390/real 1787757390] req@ffff8c05b2e70380 x1874588654914944/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757390.935073:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757390.935101:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757390.935108:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757390.935128:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10447 00000100:00080000:2.0:1787757390.935144:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757390.935157:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757390.935170:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588654914944/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787757390.935175:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787757390.935210:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787757390.935216:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588654915712/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757390.935221:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757390.935237:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654914944:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757390.935285:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654915072:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757390.935327:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757390.935354:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757390/real 1787757390] req@ffff8c05b2e71880 x1874588654915072/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757390.935372:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588654915072/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757390.935388:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757390.935420:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757390.935429:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654915072:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757390.935444:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654915200:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757390.935489:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757390.935511:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757390/real 1787757390] req@ffff8c05b2e71c00 x1874588654915200/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757390.935527:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588654915200/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757390.935539:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757390.935543:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757390.935547:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654915200:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757390.935555:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654915328:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757390.935581:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757390.935594:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757390/real 1787757390] req@ffff8c05b2e73b80 x1874588654915328/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757390.935609:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588654915328/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757390.935620:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757390.935623:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757390.935627:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654915328:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757390.935634:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654915456:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757390.935654:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757390.935665:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757390/real 1787757390] req@ffff8c05b2e71500 x1874588654915456/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757390.935680:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588654915456/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757390.935690:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757390.935693:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757390.935698:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654915456:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757390.935737:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654915584:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757390.935779:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757390.935801:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757390/real 1787757390] req@ffff8c05b2e73480 x1874588654915584/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757390.935822:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588654915584/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757390.935842:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757390.935846:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757390.935851:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654915584:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757390.935859:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654915712:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757390.935922:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757390.935937:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757390/real 1787757390] req@ffff8c05b2e70700 x1874588654915712/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757390.935952:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588654915712/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757390.935963:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757390.935967:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757390.935971:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654915712:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757391.061998:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757391.510214:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757392.086015:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757392.534163:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757393.109998:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757393.558243:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757394.133999:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757394.582152:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757395.157993:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757395.606377:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757396.054106:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757396.054121:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757396.054129:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757396.054146:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10452 00000100:00080000:3.0:1787757396.054163:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757396.054175:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757396.054240:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588654915840/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757396.054286:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757396.054292:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757396.054294:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757396.054301:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10452 00000100:00080000:3.0:1787757396.054328:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757396.054333:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757396.054349:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757396.054369:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588654915968/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757396.054384:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757396.054390:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757396.054393:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757396.054402:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10452 00000100:00080000:3.0:1787757396.054407:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757396.054411:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757396.054420:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757396.054434:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588654916096/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757396.054450:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654915840:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757396.054465:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757396.054472:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757396.054475:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757396.054483:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10452 00000100:00080000:3.0:1787757396.054488:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757396.054494:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757396.054501:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757396.054527:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588654916224/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757396.054539:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757396.054546:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757396.054548:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757396.054553:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10452 00000100:00080000:3.0:1787757396.054558:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757396.054565:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:0.0:1787757396.054574:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:3.0:1787757396.054581:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757396.054604:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588654916352/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757396.054617:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757396.054622:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757396.054626:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757396.054635:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10452 00000100:00080000:3.0:1787757396.054643:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757396.054655:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757396.054656:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757396/real 1787757396] req@ffff8c05a9f78700 x1874588654915840/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787757396.054668:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757396.054714:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588654916480/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757396.054730:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588654915840/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757396.054746:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757396.054774:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757396.054778:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757396.054781:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757396.054800:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10452 00000100:00080000:0.0:1787757396.054801:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787757396.054822:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757396.054826:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654915840:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757396.054835:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757396.054843:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757396.054846:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654915968:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757396.054933:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588654916608/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757396.054951:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757396.054958:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757396.054961:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757396.054970:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10452 00000100:00080000:3.0:1787757396.054980:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787757396.054985:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757396.055002:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757396.055019:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757396/real 1787757396] req@ffff8c05a9f7aa00 x1874588654915968/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787757396.055022:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757396.055058:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588654915968/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757396.055072:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588654916736/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757396.055077:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757396.055101:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757396.055126:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654915968:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757396.055171:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654916096:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757396.055201:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757396.055216:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757396/real 1787757396] req@ffff8c05a9f7a300 x1874588654916096/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757396.055231:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588654916096/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757396.055249:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757396.055253:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757396.055257:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654916096:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757396.055265:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654916224:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757396.055285:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757396.055297:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757396/real 1787757396] req@ffff8c05a9f79880 x1874588654916224/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757396.055347:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588654916224/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757396.055359:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757396.055363:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757396.055367:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654916224:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757396.055374:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654916352:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757396.055399:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757396.055412:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757396/real 1787757396] req@ffff8c05a9f78380 x1874588654916352/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757396.055427:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588654916352/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757396.055436:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757396.055440:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757396.055445:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654916352:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757396.055452:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654916480:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757396.055472:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757396.055483:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757396/real 1787757396] req@ffff8c05a9f78a80 x1874588654916480/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757396.055497:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588654916480/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757396.055506:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757396.055510:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757396.055514:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654916480:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757396.055520:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654916608:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757396.055539:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757396.055550:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757396/real 1787757396] req@ffff8c05a9f7a680 x1874588654916608/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757396.055563:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588654916608/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757396.055573:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757396.055577:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757396.055581:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654916608:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757396.055588:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654916736:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757396.055606:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757396.055618:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757396/real 1787757396] req@ffff8c05a9f7b480 x1874588654916736/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757396.055632:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588654916736/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757396.055641:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757396.055645:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757396.055649:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654916736:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757396.182102:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757396.630163:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757397.205997:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757397.654249:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757398.229993:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757398.678145:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757399.254046:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757399.702178:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757400.278025:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757400.726182:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757401.174011:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757401.174023:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757401.174029:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757401.174045:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10457 00000100:00080000:3.0:1787757401.174062:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757401.174074:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757401.174136:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588654916864/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757401.174182:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757401.174190:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757401.174194:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757401.174203:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10457 00000100:00080000:3.0:1787757401.174209:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757401.174245:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757401.174259:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757401.174277:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654916864:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757401.174286:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588654916992/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757401.174305:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757401.174311:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757401.174313:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757401.174320:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10457 00000100:00080000:3.0:1787757401.174327:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757401.174332:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757401.174339:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757401.174354:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588654917120/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757401.174368:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757401.174372:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757401.174375:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757401.174379:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10457 00000100:00080000:3.0:1787757401.174384:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757401.174388:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757401.174395:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757401.174408:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588654917248/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757401.174420:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757401.174424:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757401.174427:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757401.174432:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10457 00000100:00080000:3.0:1787757401.174437:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757401.174443:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757401.174451:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757401.174455:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654916992:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757401.174472:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588654917376/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757401.174487:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757401.174492:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757401.174495:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757401.174503:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10457 00000100:00080000:3.0:1787757401.174509:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757401.174518:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654917120:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757401.174520:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757401.174539:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757401.174563:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654917248:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757401.174570:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588654917504/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757401.174596:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787757401.174605:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654917376:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757401.174607:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757401.174617:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757401.174631:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10457 00000100:00080000:3.0:1787757401.174640:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757401.174644:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654917504:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757401.174654:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757401.174665:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757401.174689:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588654917632/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757401.174716:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757401.174721:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757401.174724:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757401.174728:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10457 00000100:00080000:3.0:1787757401.174734:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757401.174741:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757401.174748:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757401.174760:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654917632:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757401.174768:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588654917760/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757401.174810:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654917760:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757401.301956:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757401.750288:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757402.326030:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757402.774167:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757403.349988:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757403.798257:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787757404.246422:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.107:988 00000400:00000100:0.0:1787757404.246479:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.107@tcp at host 192.168.201.107 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787757404.246501:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787757404.246562:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787757404.246578:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787757404.246592:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787757404.246613:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787757404.246633:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787757404.246660:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787757404.246677:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:3.0:1787757404.246695:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757401/real 1787757404] req@ffff8c05a9f79880 x1874588654917376/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757404.246749:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588654917376/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757404.246772:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757404.246781:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757404.246790:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654917376:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757404.246806:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757401/real 1787757404] req@ffff8c05a9f7a300 x1874588654917504/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757404.246834:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588654917504/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757404.246851:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757404.246858:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757404.246865:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654917504:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757404.246950:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757401/real 1787757404] req@ffff8c05a9f7a680 x1874588654916992/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757404.247003:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588654916992/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757404.247020:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757404.247027:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757404.247033:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654916992:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757404.247044:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757401/real 1787757404] req@ffff8c05a9f78a80 x1874588654917120/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757404.247066:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588654917120/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757404.247081:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757404.247088:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757404.247094:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654917120:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757404.247107:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757401/real 1787757404] req@ffff8c05a9f78380 x1874588654917248/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757404.247133:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588654917248/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757404.247147:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757404.247154:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757404.247190:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654917248:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757404.247205:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757401/real 1787757404] req@ffff8c05a9f7b480 x1874588654916864/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757404.247229:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588654916864/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757404.247244:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757404.247253:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757404.247491:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654916864:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757404.247554:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757401/real 1787757404] req@ffff8c05a9f78700 x1874588654917760/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757404.247583:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588654917760/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757404.247634:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757404.247643:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757404.247650:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654917760:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757404.247687:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757401/real 1787757404] req@ffff8c05a9f7aa00 x1874588654917632/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757404.247718:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588654917632/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757404.247734:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757404.247741:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757404.247747:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654917632:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757404.374033:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757404.822075:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757405.398075:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757405.846219:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757406.294002:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757406.294016:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757406.294021:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757406.294037:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10462 00000100:00080000:3.0:1787757406.294051:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757406.294061:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757406.294108:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588654917888/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757406.294188:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757406.294195:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757406.294199:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757406.294216:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10462 00000100:00080000:3.0:1787757406.294224:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757406.294233:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757406.294248:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757406.294274:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588654918016/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757406.294295:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757406.294303:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787757406.294305:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654917888:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757406.294306:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757406.294313:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10462 00000100:00080000:3.0:1787757406.294320:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757406.294326:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757406.294335:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757406.294354:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588654918144/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757406.294372:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757406.294377:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757406.294381:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757406.294387:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10462 00000100:00080000:3.0:1787757406.294393:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757406.294400:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757406.294409:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787757406.294415:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757406.294427:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588654918272/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757406.294444:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757406.294450:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757406.294453:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757406.294459:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10462 00000100:00080000:3.0:1787757406.294465:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757406.294472:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757406.294477:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757406/real 1787757406] req@ffff8c05a9f7aa00 x1874588654917888/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787757406.294481:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757406.294499:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588654918400/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757406.294509:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588654917888/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757406.294517:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757406.294524:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757406.294527:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757406.294529:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757406.294535:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10462 00000100:00080000:0.0:1787757406.294536:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787757406.294540:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757406.294544:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654917888:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757406.294547:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757406.294555:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757406.294572:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588654918528/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757406.294589:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757406.294595:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787757406.294598:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654918016:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757406.294599:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757406.294606:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10462 00000100:00080000:3.0:1787757406.294612:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757406.294619:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757406.294628:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787757406.294639:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757406.294647:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588654918656/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757406.294663:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757406/real 1787757406] req@ffff8c05a9f78700 x1874588654918016/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757406.294665:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757406.294671:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757406.294675:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757406.294682:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10462 00000100:00080000:3.0:1787757406.294687:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757406.294694:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757406.294703:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757406.294710:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588654918016/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757406.294723:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588654918784/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757406.294728:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757406.294735:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757406.294742:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654918016:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757406.294753:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654918144:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757406.294796:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757406.294821:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757406/real 1787757406] req@ffff8c05a9f7b480 x1874588654918144/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757406.294847:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588654918144/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757406.294864:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757406.294871:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757406.294954:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654918144:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757406.294969:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654918272:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757406.295013:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757406.295037:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757406/real 1787757406] req@ffff8c05a9f78380 x1874588654918272/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757406.295063:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588654918272/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757406.295079:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757406.295086:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757406.295092:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654918272:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757406.295105:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654918400:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757406.295176:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757406.295201:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757406/real 1787757406] req@ffff8c05a9f78a80 x1874588654918400/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757406.295226:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588654918400/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757406.295243:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757406.295249:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757406.295257:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654918400:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757406.295269:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654918528:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757406.295311:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757406.295335:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757406/real 1787757406] req@ffff8c05a9f7a680 x1874588654918528/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757406.295358:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588654918528/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757406.295388:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757406.295395:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757406.295402:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654918528:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757406.295442:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654918656:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757406.295487:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757406.295510:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757406/real 1787757406] req@ffff8c05a9f7a300 x1874588654918656/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757406.295532:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588654918656/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757406.295547:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757406.295553:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757406.295560:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654918656:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757406.295571:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654918784:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757406.295611:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757406.295634:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757406/real 1787757406] req@ffff8c05a9f79880 x1874588654918784/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757406.295659:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588654918784/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757406.295676:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757406.295683:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757406.295691:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654918784:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757406.422034:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757406.870105:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757407.445997:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757407.894272:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757408.470005:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757408.918159:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787757409.493985:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757409.942225:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757410.518015:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757410.966125:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757411.414062:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757411.414076:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787757411.414082:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757411.414095:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10467 00000100:00080000:2.0:1787757411.414108:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757411.414117:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757411.414168:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588654918912/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757411.414209:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757411.414214:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757411.414216:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757411.414222:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10467 00000100:00080000:2.0:1787757411.414226:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757411.414230:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757411.414241:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757411.414257:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588654919040/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757411.414282:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757411.414287:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757411.414290:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757411.414299:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10467 00000100:00080000:2.0:1787757411.414303:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757411.414307:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757411.414313:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757411.414315:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654918912:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787757411.414328:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588654919168/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757411.414345:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757411.414350:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757411.414352:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757411.414357:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10467 00000100:00080000:2.0:1787757411.414362:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757411.414368:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757411.414375:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757411.414390:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588654919296/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757411.414405:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787757411.414406:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757411.414416:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757411.414420:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757411.414428:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10467 00000100:00080000:2.0:1787757411.414436:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757411.414441:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757411.414447:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757411.414467:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588654919424/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757411.414476:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757411/real 1787757411] req@ffff8c05b2e73100 x1874588654918912/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757411.414484:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757411.414498:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757411.414505:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757411.414516:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10467 00000100:00080000:2.0:1787757411.414521:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757411.414526:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588654918912/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757411.414531:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757411.414544:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787757411.414555:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757411.414566:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757411.414567:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588654919552/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757411.414576:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654918912:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757411.414600:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787757411.414604:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654919040:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757411.414609:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787757411.414618:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757411.414625:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10467 00000100:00080000:2.0:1787757411.414630:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757411.414635:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757411.414642:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787757411.414650:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757411.414663:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588654919680/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757411.414685:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787757411.414690:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757411/real 1787757411] req@ffff8c05b2e72d80 x1874588654919040/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757411.414692:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757411.414699:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757411.414713:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10467 00000100:00080000:2.0:1787757411.414725:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757411.414733:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588654919040/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757411.414734:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757411.414748:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787757411.414765:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787757411.414772:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588654919808/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757411.414773:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757411.414785:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654919040:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757411.414838:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654919168:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757411.414870:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757411.414929:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757411/real 1787757411] req@ffff8c05b2e72680 x1874588654919168/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757411.414946:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588654919168/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757411.414957:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757411.414961:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757411.414965:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654919168:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757411.414973:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654919296:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757411.414998:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757411.415010:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757411/real 1787757411] req@ffff8c05b2e70e00 x1874588654919296/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757411.415023:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588654919296/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757411.415059:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757411.415064:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757411.415068:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654919296:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757411.415075:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654919424:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757411.415099:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757411.415113:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757411/real 1787757411] req@ffff8c05b2e70000 x1874588654919424/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757411.415148:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588654919424/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757411.415159:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757411.415163:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757411.415166:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654919424:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757411.415173:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654919552:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757411.415195:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757411.415207:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757411/real 1787757411] req@ffff8c05b2e72300 x1874588654919552/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757411.415221:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588654919552/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757411.415230:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757411.415233:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757411.415237:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654919552:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757411.415244:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654919680:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757411.415263:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757411.415274:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757411/real 1787757411] req@ffff8c05b2e72a00 x1874588654919680/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757411.415288:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588654919680/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757411.415305:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757411.415309:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757411.415313:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654919680:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757411.415319:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654919808:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757411.415337:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757411.415347:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757411/real 1787757411] req@ffff8c05b2e71180 x1874588654919808/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757411.415360:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588654919808/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757411.415370:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757411.415374:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757411.415377:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654919808:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757411.542073:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757411.990247:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757412.566066:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757413.014170:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757413.590046:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757414.038214:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757414.614003:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757415.062098:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757415.637972:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757416.086085:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757416.534120:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757416.534135:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757416.534141:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757416.534159:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10472 00000100:00080000:3.0:1787757416.534176:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757416.534187:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757416.534240:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588654919936/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757416.534279:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757416.534284:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757416.534286:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757416.534290:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10472 00000100:00080000:3.0:1787757416.534294:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757416.534300:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757416.534317:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757416.534347:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588654920064/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757416.534360:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654919936:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757416.534378:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757416.534387:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757416.534391:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757416.534402:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10472 00000100:00080000:3.0:1787757416.534409:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757416.534415:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757416.534423:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757416.534445:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588654920192/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757416.534460:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757416.534465:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757416.534467:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757416.534471:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10472 00000100:00080000:3.0:1787757416.534476:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757416.534479:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:2.0:1787757416.534491:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757416.534566:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757416/real 1787757416] req@ffff8c05a9f7ad80 x1874588654919936/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757416.534602:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588654919936/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787757416.534604:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787757416.534645:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757416.534656:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757416.534684:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654919936:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787757416.534735:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654920064:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757416.534778:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757416.534812:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757416/real 1787757416] req@ffff8c05a9f7bb80 x1874588654920064/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757416.534837:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588654920064/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757416.534853:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757416.534867:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757416.535000:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654920064:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757416.535022:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654920192:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757416.535079:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757416.535082:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588654920320/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757416.535105:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757416/real 1787757416] req@ffff8c05a9f79180 x1874588654920192/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757416.535109:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757416.535135:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757416.535146:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757416.535158:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10472 00000100:00100000:2.0:1787757416.535160:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588654920192/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757416.535173:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757416.535197:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:2.0:1787757416.535200:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757416.535214:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000008:00100000:3.0:1787757416.535214:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757416.535229:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654920192:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757416.535244:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588654920448/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757416.535262:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654920320:192.168.201.107@tcp:8:kworker.0 00000100:00080000:3.0:1787757416.535262:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757416.535288:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757416.535291:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757416.535297:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10472 00000100:00080000:3.0:1787757416.535315:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787757416.535323:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757416.535336:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757416.535345:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757416/real 1787757416] req@ffff8c05a9f79f80 x1874588654920320/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787757416.535353:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757416.535369:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588654920320/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757416.535390:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588654920576/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757416.535486:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757416.535489:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757416.535498:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787757416.535506:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787757416.535508:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787757416.535536:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654920320:192.168.201.107@tcp:8:kworker.0 00000100:00080000:3.0:1787757416.535545:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10472 00000100:00080000:3.0:1787757416.535565:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757416.535575:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757416.535588:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757416.535591:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654920448:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757416.535635:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588654920704/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787757416.535661:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757416.535667:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757416.535674:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757416.535677:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757416.535685:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10472 00000100:00080000:3.0:1787757416.535692:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757416.535703:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757416.535706:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757416/real 1787757416] req@ffff8c05a9f78e00 x1874588654920448/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787757416.535712:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757416.535747:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588654920832/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757416.535756:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588654920448/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757416.535816:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757416.535824:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757416.535831:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654920448:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757416.535847:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654920576:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757416.535967:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757416.536019:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757416/real 1787757416] req@ffff8c05a9f79500 x1874588654920576/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757416.536055:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588654920576/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757416.536070:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757416.536074:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757416.536079:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654920576:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757416.536104:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654920704:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757416.536135:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757416.536155:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757416/real 1787757416] req@ffff8c05a9f7b100 x1874588654920704/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757416.536170:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588654920704/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757416.536180:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757416.536184:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757416.536188:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654920704:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757416.536195:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654920832:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757416.536213:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757416.536224:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757416/real 1787757416] req@ffff8c05a9f7b800 x1874588654920832/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757416.536238:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588654920832/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757416.536248:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757416.536251:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757416.536255:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654920832:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757416.662082:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757417.110016:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757417.686057:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757418.134284:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757418.710094:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757419.158108:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757419.733972:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757420.182100:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757420.757949:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757421.205991:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757421.654053:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757421.654070:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787757421.654078:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757421.654094:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10478 00000100:00080000:2.0:1787757421.654111:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757421.654145:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757421.654201:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588654920960/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757421.654251:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757421.654268:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757421.654273:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757421.654281:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10478 00000100:00080000:2.0:1787757421.654289:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757421.654299:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757421.654316:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757421.654351:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588654921088/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757421.654359:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654920960:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757421.654398:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757421.654407:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757421.654412:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757421.654424:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10478 00000100:00080000:2.0:1787757421.654434:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757421.654443:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757421.654455:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787757421.654472:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757421.654478:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588654921216/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757421.654505:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757421.654509:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757421.654513:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757421.654522:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10478 00000100:00080000:2.0:1787757421.654531:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757421.654538:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757421/real 1787757421] req@ffff8c05b2e70a80 x1874588654920960/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757421.654539:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757421.654555:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757421.654574:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588654920960/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757421.654584:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588654921344/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757421.654591:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757421.654604:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757421.654617:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654920960:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757421.654643:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757421.654652:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757421.654658:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757421.654664:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654921088:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757421.654673:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10478 00000100:00080000:2.0:1787757421.654686:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787757421.654697:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757421.654699:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757421.654714:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757421/real 1787757421] req@ffff8c05b2e71180 x1874588654921088/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787757421.654717:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757421.654742:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588654921088/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757421.654751:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757421.654755:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757421.654766:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654921088:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757421.654768:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588654921472/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757421.654788:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654921216:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757421.654809:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757421.654814:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757421.654818:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787757421.654828:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757421.654834:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10478 00000100:00080000:2.0:1787757421.654843:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757421.654849:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757421/real 1787757421] req@ffff8c05b2e72a00 x1874588654921216/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757421.654908:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588654921216/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757421.654915:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757421.654921:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757421.654925:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654921216:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757421.654932:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654921344:192.168.201.107@tcp:8:kworker.0 00000100:00080000:2.0:1787757421.654937:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757421.654949:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787757421.654962:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757421.654978:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588654921600/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757421.654987:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757421/real 1787757421] req@ffff8c05b2e72300 x1874588654921344/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757421.655041:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787757421.655049:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588654921344/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757421.655052:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787757421.655060:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757421.655078:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757421.655078:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10478 00000100:00080000:0.0:1787757421.655097:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787757421.655098:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757421.655120:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654921344:192.168.201.107@tcp:8:kworker.0 00000100:00080000:2.0:1787757421.655121:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757421.655142:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757421.655161:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654921472:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757421.655163:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588654921728/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757421.655202:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757421.655206:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757421.655209:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787757421.655212:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757421.655222:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10478 00000100:00080000:2.0:1787757421.655231:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757421.655235:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757421/real 1787757421] req@ffff8c05b2e70000 x1874588654921472/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757421.655243:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757421.655256:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757421.655263:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588654921472/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757421.655278:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757421.655286:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757421.655289:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588654921856/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757421.655291:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654921472:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757421.655309:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654921600:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757421.655339:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757421.655354:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757421/real 1787757421] req@ffff8c05b2e70e00 x1874588654921600/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757421.655379:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588654921600/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757421.655385:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757421.655388:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757421.655390:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654921600:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757421.655407:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654921728:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757421.655421:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757421.655427:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757421/real 1787757421] req@ffff8c05b2e72680 x1874588654921728/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757421.655434:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588654921728/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757421.655439:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757421.655441:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757421.655444:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654921728:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757421.655448:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654921856:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757421.655460:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757421.655466:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757421/real 1787757421] req@ffff8c05b2e72d80 x1874588654921856/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757421.655473:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588654921856/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757421.655479:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757421.655480:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757421.655482:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654921856:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757421.782021:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757422.230181:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757422.806030:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757423.254094:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757423.829996:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757424.278190:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757424.853999:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757425.302147:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757425.878007:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757426.326229:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757426.774056:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757426.774073:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757426.774080:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757426.774106:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10483 00000100:00080000:3.0:1787757426.774122:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757426.774134:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757426.774192:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588654921984/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757426.774245:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757426.774252:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757426.774257:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757426.774266:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10483 00000100:00080000:3.0:1787757426.774274:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757426.774284:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757426.774301:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757426.774325:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654921984:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757426.774329:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588654922112/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757426.774355:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757426.774359:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757426.774363:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757426.774368:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10483 00000100:00080000:3.0:1787757426.774374:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757426.774379:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757426.774386:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757426.774401:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588654922240/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757426.774414:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757426.774419:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757426.774425:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787757426.774427:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757426.774435:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10483 00000100:00080000:3.0:1787757426.774443:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757426.774449:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757426.774456:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757426.774472:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588654922368/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757426.774484:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757426.774488:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757426.774491:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757426.774496:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10483 00000100:00080000:3.0:1787757426.774502:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757426.774507:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757426.774514:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757426.774516:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757426/real 1787757426] req@ffff8c05a9f7b800 x1874588654921984/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757426.774534:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588654922496/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757426.774558:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787757426.774565:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588654921984/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757426.774568:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757426.774575:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757426.774592:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10483 00000100:00080000:3.0:1787757426.774604:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757426.774607:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757426.774617:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787757426.774626:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000008:00100000:3.0:1787757426.774633:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757426.774641:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654921984:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757426.774668:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588654922624/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757426.774685:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654922112:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757426.774698:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757426.774707:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757426.774712:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757426.774724:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10483 00000100:00080000:3.0:1787757426.774734:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787757426.774735:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757426.774752:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757426.774796:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757426/real 1787757426] req@ffff8c05a9f7b100 x1874588654922112/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757426.774819:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588654922112/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787757426.774819:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787757426.774844:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757426.774849:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757426.774860:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654922112:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757426.774863:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588654922752/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757426.774873:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654922240:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757426.774958:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757426.774960:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757426.774968:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757426.774972:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757426.774981:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10483 00000100:00100000:0.0:1787757426.774986:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757426/real 1787757426] req@ffff8c05a9f79500 x1874588654922240/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757426.774991:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757426.775006:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757426.775018:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757426.775021:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588654922240/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757426.775037:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787757426.775045:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588654922880/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757426.775049:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757426.775064:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654922240:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757426.775084:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654922368:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757426.775115:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757426.775131:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757426/real 1787757426] req@ffff8c05a9f78e00 x1874588654922368/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757426.775145:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588654922368/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757426.775156:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757426.775160:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757426.775164:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654922368:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757426.775175:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654922496:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757426.775210:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757426.775233:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757426/real 1787757426] req@ffff8c05a9f79f80 x1874588654922496/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757426.775259:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588654922496/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757426.775276:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757426.775283:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757426.775288:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654922496:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757426.775317:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654922624:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757426.775347:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757426.775362:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757426/real 1787757426] req@ffff8c05a9f79180 x1874588654922624/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757426.775378:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588654922624/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757426.775388:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757426.775392:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757426.775397:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654922624:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757426.775404:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654922752:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757426.775423:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757426.775434:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757426/real 1787757426] req@ffff8c05a9f7bb80 x1874588654922752/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757426.775448:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588654922752/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757426.775458:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757426.775461:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757426.775465:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654922752:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757426.775471:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654922880:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757426.775489:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757426.775500:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757426/real 1787757426] req@ffff8c05a9f7ad80 x1874588654922880/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757426.775514:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588654922880/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757426.775524:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757426.775527:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757426.775531:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654922880:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757426.902013:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757427.350120:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757427.926037:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757428.374315:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757428.950060:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757429.398163:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757429.974001:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757430.422221:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757430.998019:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757431.446154:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757431.894106:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757431.894124:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787757431.894131:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757431.894149:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10488 00000100:00080000:1.0:1787757431.894168:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757431.894179:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757431.894248:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588654923008/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757431.894292:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757431.894297:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757431.894300:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757431.894305:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10488 00000100:00080000:1.0:1787757431.894309:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757431.894314:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757431.894328:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757431.894346:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588654923136/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757431.894361:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757431.894365:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757431.894369:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757431.894374:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10488 00000100:00080000:1.0:1787757431.894379:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757431.894384:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757431.894390:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757431.894407:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588654923264/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757431.894420:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757431.894425:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757431.894428:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757431.894433:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10488 00000100:00080000:1.0:1787757431.894436:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757431.894440:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757431.894446:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757431.894462:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654923008:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787757431.894465:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654923392/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757431.894512:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757431.894521:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757431.894525:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757431.894536:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10488 00000100:00080000:1.0:1787757431.894545:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757431.894555:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757431.894568:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787757431.894587:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757431.894604:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654923520/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757431.894645:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757431.894650:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757431.894661:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757431.894662:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757431/real 1787757431] req@ffff8c05b2d9bb80 x1874588654923008/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757431.894728:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10488 00000100:00080000:1.0:1787757431.894739:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757431.894751:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588654923008/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757431.894762:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787757431.894781:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787757431.894786:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787757431.894794:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757431.894807:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654923008:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787757431.894823:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588654923648/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757431.894826:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654923136:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787757431.894869:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757431.894935:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787757431.894939:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757431.894948:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10488 00000100:00080000:1.0:1787757431.894974:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757431.894985:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:0.0:1787757431.894994:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:1.0:1787757431.895003:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757431.895027:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757431/real 1787757431] req@ffff8c05b2d9aa00 x1874588654923136/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757431.895029:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654923776/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757431.895153:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757431.895159:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757431.895168:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757431.895170:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588654923136/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757431.895196:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10488 00000100:00080000:1.0:1787757431.895216:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757431.895219:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757431.895234:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787757431.895238:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000008:00100000:1.0:1787757431.895250:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757431.895257:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654923136:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757431.895281:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654923264:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757431.895283:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654923904/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757431.895338:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757431.895372:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757431/real 1787757431] req@ffff8c05b2d9b100 x1874588654923264/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757431.895412:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654923264/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757431.895430:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757431.895438:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757431.895444:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654923264:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757431.895488:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654923392:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757431.895521:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757431.895536:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757431/real 1787757431] req@ffff8c05b2d99880 x1874588654923392/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757431.895552:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654923392/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757431.895562:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757431.895567:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757431.895571:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654923392:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757431.895578:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654923520:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757431.895598:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757431.895610:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757431/real 1787757431] req@ffff8c05b2d98380 x1874588654923520/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757431.895624:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654923520/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757431.895641:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757431.895645:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757431.895649:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654923520:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757431.895656:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654923648:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757431.895707:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757431.895726:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757431/real 1787757431] req@ffff8c05b2d9b480 x1874588654923648/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757431.895741:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654923648/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757431.895751:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757431.895755:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757431.895759:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654923648:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757431.895767:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654923776:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757431.895792:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757431.895804:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757431/real 1787757431] req@ffff8c05b2d98000 x1874588654923776/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757431.895819:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654923776/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757431.895828:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757431.895831:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757431.895836:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654923776:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757431.895844:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654923904:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757431.895863:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757431.895874:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757431/real 1787757431] req@ffff8c05b2d99f80 x1874588654923904/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757431.895925:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654923904/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757431.895936:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757431.895940:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757431.895944:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654923904:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757432.022007:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757432.470276:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757433.046021:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757433.494085:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757434.070002:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757434.518251:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757435.094030:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757435.542136:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757436.118016:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757436.566211:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757437.014124:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757437.014144:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757437.014153:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757437.014174:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10493 00000100:00080000:3.0:1787757437.014196:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757437.014212:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757437.014285:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588654924032/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757437.014329:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757437.014335:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757437.014338:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757437.014344:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10493 00000100:00080000:3.0:1787757437.014350:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757437.014355:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757437.014370:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757437.014389:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588654924160/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757437.014405:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757437.014410:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757437.014413:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757437.014425:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10493 00000100:00100000:0.0:1787757437.014429:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654924032:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757437.014430:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757437.014444:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757437.014452:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757437.014469:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588654924288/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757437.014483:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757437.014488:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757437.014490:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757437.014496:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10493 00000100:00080000:3.0:1787757437.014499:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757437.014504:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757437.014512:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757437.014525:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588654924416/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757437.014538:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757437.014542:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757437.014544:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757437.014549:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10493 00000100:00080000:3.0:1787757437.014554:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787757437.014556:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757437.014565:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757437.014574:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757437.014638:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588654924544/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757437.014653:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757437.014662:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757437.014666:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757437.014677:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10493 00000100:00100000:0.0:1787757437.014678:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757437/real 1787757437] req@ffff8c05a9f79880 x1874588654924032/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757437.014687:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757437.014706:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757437.014720:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757437.014726:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588654924032/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757437.014748:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787757437.014755:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588654924672/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757437.014760:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757437.014776:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654924032:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757437.014788:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757437.014795:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757437.014798:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757437.014806:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10493 00000100:00080000:3.0:1787757437.014812:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757437.014819:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757437.014827:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757437.014828:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654924160:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757437.014854:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588654924800/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757437.014871:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757437.014919:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757437.014922:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757437.014929:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10493 00000100:00080000:3.0:1787757437.014935:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787757437.014946:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757437.014947:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757437.014964:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757437.014988:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588654924928/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757437.014993:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757437/real 1787757437] req@ffff8c05a9f7a300 x1874588654924160/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757437.015044:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588654924160/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757437.015058:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757437.015066:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757437.015071:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654924160:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757437.015080:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654924288:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757437.015135:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757437.015161:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757437/real 1787757437] req@ffff8c05a9f7a680 x1874588654924288/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757437.015187:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588654924288/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757437.015205:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757437.015211:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757437.015216:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654924288:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757437.015227:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654924416:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757437.015271:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757437.015296:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757437/real 1787757437] req@ffff8c05a9f78a80 x1874588654924416/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757437.015336:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588654924416/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757437.015354:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757437.015360:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757437.015367:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654924416:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757437.015379:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654924544:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757437.015423:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757437.015447:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757437/real 1787757437] req@ffff8c05a9f78380 x1874588654924544/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757437.015470:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588654924544/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757437.015481:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757437.015485:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757437.015489:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654924544:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757437.015497:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654924672:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757437.015525:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757437.015538:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757437/real 1787757437] req@ffff8c05a9f7b480 x1874588654924672/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757437.015557:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588654924672/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757437.015610:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757437.015619:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757437.015626:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654924672:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757437.015668:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654924800:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757437.015704:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757437.015721:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757437/real 1787757437] req@ffff8c05a9f78700 x1874588654924800/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757437.015736:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588654924800/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757437.015748:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757437.015752:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757437.015756:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654924800:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757437.015762:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654924928:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757437.015783:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757437.015795:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757437/real 1787757437] req@ffff8c05a9f7aa00 x1874588654924928/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757437.015810:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588654924928/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757437.015819:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757437.015822:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757437.015827:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654924928:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757437.142031:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757437.590145:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757438.165981:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757438.614214:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757439.190027:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757439.638184:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757440.214008:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757440.662164:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757441.238082:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757441.686144:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757442.133993:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757442.134007:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757442.134014:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757442.134027:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10498 00000100:00080000:3.0:1787757442.134041:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757442.134051:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757442.134101:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588654925056/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757442.134137:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757442.134142:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757442.134145:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757442.134151:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10498 00000100:00080000:3.0:1787757442.134157:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757442.134162:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757442.134172:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757442.134189:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588654925184/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757442.134200:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757442.134205:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757442.134208:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757442.134218:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10498 00000100:00100000:0.0:1787757442.134221:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654925056:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757442.134225:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757442.134237:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757442.134246:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757442.134267:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588654925312/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757442.134285:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757442.134291:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757442.134293:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757442.134299:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10498 00000100:00080000:3.0:1787757442.134305:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757442.134313:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:0.0:1787757442.134317:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:3.0:1787757442.134322:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757442.134339:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588654925440/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757442.134349:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757442.134353:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757442.134355:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757442.134369:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10498 00000100:00080000:3.0:1787757442.134375:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757442.134380:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757442.134385:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757442/real 1787757442] req@ffff8c05a9f7aa00 x1874588654925056/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787757442.134388:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757442.134407:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588654925568/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757442.134418:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588654925056/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757442.134420:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757442.134438:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757442.134445:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787757442.134446:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757442.134452:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757442.134458:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654925056:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757442.134463:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10498 00000100:00080000:3.0:1787757442.134471:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757442.134477:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654925184:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757442.134479:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757442.134510:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757442.134532:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588654925696/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757442.134555:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757442.134555:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757442.134563:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757442.134565:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757442.134572:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10498 00000100:00080000:3.0:1787757442.134577:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757442.134579:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757442/real 1787757442] req@ffff8c05a9f78700 x1874588654925184/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757442.134586:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757442.134595:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757442.134607:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588654925184/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757442.134613:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588654925824/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757442.134630:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757442.134643:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757442.134655:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654925184:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757442.134656:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757442.134666:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757442.134670:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757442.134676:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10498 00000100:00080000:3.0:1787757442.134680:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757442.134684:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757442.134688:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757442.134698:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654925312:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757442.134703:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588654925952/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757442.134730:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757442.134746:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757442/real 1787757442] req@ffff8c05a9f7b480 x1874588654925312/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757442.134757:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588654925312/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757442.134765:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757442.134768:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757442.134771:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654925312:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757442.134777:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654925440:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757442.134791:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757442.134800:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757442/real 1787757442] req@ffff8c05a9f78380 x1874588654925440/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757442.134810:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588654925440/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757442.134818:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757442.134821:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757442.134823:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654925440:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757442.134829:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654925568:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757442.134843:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757442.134852:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757442/real 1787757442] req@ffff8c05a9f78a80 x1874588654925568/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757442.134869:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588654925568/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757442.134922:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757442.134927:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757442.134932:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654925568:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757442.134989:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654925696:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757442.135045:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757442.135064:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757442/real 1787757442] req@ffff8c05a9f7a680 x1874588654925696/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757442.135079:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588654925696/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757442.135090:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757442.135095:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757442.135099:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654925696:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757442.135105:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654925824:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757442.135125:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757442.135135:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757442/real 1787757442] req@ffff8c05a9f7a300 x1874588654925824/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757442.135146:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588654925824/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757442.135162:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757442.135165:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757442.135168:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654925824:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757442.135188:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654925952:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757442.135206:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757442.135223:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757442/real 1787757442] req@ffff8c05a9f79880 x1874588654925952/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757442.135258:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588654925952/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757442.135270:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757442.135276:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757442.135281:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654925952:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757442.262006:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757442.710220:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757443.286023:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757443.734100:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757444.309977:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757444.758229:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757445.334008:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757445.782150:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757446.358030:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757446.806261:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757447.254097:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757447.254113:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787757447.254121:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757447.254137:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10503 00000100:00080000:1.0:1787757447.254155:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757447.254166:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757447.254233:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588654926080/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757447.254283:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757447.254288:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757447.254291:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757447.254296:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10503 00000100:00080000:1.0:1787757447.254300:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757447.254305:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757447.254320:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757447.254353:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588654926208/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757447.254367:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757447.254372:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757447.254374:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757447.254378:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10503 00000100:00080000:1.0:1787757447.254383:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757447.254389:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757447.254397:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757447.254444:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654926336/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757447.254461:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757447.254470:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787757447.254472:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654926080:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757447.254473:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757447.254487:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10503 00000100:00080000:1.0:1787757447.254495:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757447.254500:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757447.254508:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757447.254527:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654926464/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757447.254541:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757447.254546:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757447.254550:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757447.254555:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10503 00000100:00080000:1.0:1787757447.254588:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757447.254593:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757447.254600:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787757447.254611:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757447.254618:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588654926592/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757447.254633:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757447.254638:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757447.254641:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757447.254647:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10503 00000100:00080000:1.0:1787757447.254662:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757447.254666:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757447.254673:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757447.254688:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654926720/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757447.254699:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757447.254708:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787757447.254710:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757447/real 1787757447] req@ffff8c05b2d98700 x1874588654926080/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757447.254712:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757447.254728:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10503 00000100:00080000:1.0:1787757447.254739:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757447.254750:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757447.254763:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757447.254784:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654926848/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757447.254791:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654926080/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757447.254810:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757447.254827:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757447.254827:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757447.254834:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757447.254837:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787757447.254858:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10503 00000100:00100000:0.0:1787757447.254859:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654926080:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757447.254967:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757447.254981:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654926208:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787757447.254982:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757447.255000:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757447.255030:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588654926976/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757447.255046:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757447.255074:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757447/real 1787757447] req@ffff8c05b2d9a300 x1874588654926208/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757447.255093:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588654926208/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757447.255105:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757447.255111:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757447.255115:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654926208:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757447.255152:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654926336:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757447.255176:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757447.255189:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757447/real 1787757447] req@ffff8c05b2d99f80 x1874588654926336/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757447.255204:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654926336/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757447.255222:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757447.255227:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757447.255231:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654926336:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757447.255238:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654926464:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757447.255257:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757447.255362:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757447/real 1787757447] req@ffff8c05b2d98000 x1874588654926464/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757447.255381:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654926464/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757447.255392:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757447.255396:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757447.255424:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654926464:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757447.255437:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654926592:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757447.255479:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757447.255498:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757447/real 1787757447] req@ffff8c05b2d9b480 x1874588654926592/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757447.255520:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654926592/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757447.255537:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757447.255544:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757447.255552:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654926592:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757447.255565:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654926720:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757447.255606:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757447.255630:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757447/real 1787757447] req@ffff8c05b2d98380 x1874588654926720/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757447.255646:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654926720/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757447.255657:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757447.255661:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757447.255665:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654926720:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757447.255673:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654926848:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757447.255698:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757447.255710:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757447/real 1787757447] req@ffff8c05b2d99880 x1874588654926848/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757447.255724:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654926848/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757447.255734:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757447.255738:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757447.255741:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654926848:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757447.255748:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654926976:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757447.255768:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757447.255780:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757447/real 1787757447] req@ffff8c05b2d9b100 x1874588654926976/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757447.255794:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654926976/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757447.255804:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757447.255808:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757447.255811:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654926976:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757447.382016:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757447.830143:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757448.405991:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757448.854105:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757449.429947:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757449.878066:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757450.454000:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757450.902243:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757451.477989:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757451.926088:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757452.374024:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757452.374037:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787757452.374042:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757452.374055:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10508 00000100:00080000:1.0:1787757452.374070:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757452.374082:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757452.374142:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588654927104/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757452.374211:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757452.374219:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757452.374223:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757452.374232:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10508 00000100:00080000:1.0:1787757452.374240:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757452.374248:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757452.374263:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757452.374289:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588654927232/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757452.374307:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757452.374337:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1787757452.374349:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654927104:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757452.374351:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757452.374368:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10508 00000100:00080000:1.0:1787757452.374375:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757452.374381:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757452.374391:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757452.374418:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588654927360/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757452.374444:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757452.374452:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757452.374456:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757452.374465:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10508 00000100:00080000:1.0:1787757452.374473:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757452.374481:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757452.374491:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787757452.374511:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757452.374520:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654927488/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757452.374535:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757452.374541:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757452.374544:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757452.374548:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10508 00000100:00080000:1.0:1787757452.374553:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757452.374557:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757452.374564:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757452.374577:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654927616/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757452.374587:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757452.374591:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757452.374593:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757452.374597:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10508 00000100:00080000:1.0:1787757452.374600:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757452.374604:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757452.374609:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757452.374619:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588654927744/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757452.374631:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757452.374635:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787757452.374637:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757452.374645:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10508 00000100:00080000:1.0:1787757452.374651:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757452.374657:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757452.374664:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757452.374680:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654927872/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757452.374691:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757452.374696:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757452.374698:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757452.374702:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10508 00000100:00080000:1.0:1787757452.374706:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757452.374710:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757452.374714:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757452.374728:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654928000/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757452.374853:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757452/real 1787757452] req@ffff8c05b2d9aa00 x1874588654927104/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757452.375035:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588654927104/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757452.375052:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757452.375058:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757452.375063:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654927104:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757452.375074:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654927232:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757452.375111:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757452.375130:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757452/real 1787757452] req@ffff8c05b2d9bb80 x1874588654927232/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757452.375146:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588654927232/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757452.375157:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757452.375161:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757452.375165:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654927232:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757452.375194:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654927360:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757452.375215:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757452.375227:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757452/real 1787757452] req@ffff8c05b2d9b100 x1874588654927360/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757452.375241:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654927360/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757452.375251:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757452.375256:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757452.375259:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654927360:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757452.375266:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654927488:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757452.375285:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757452.375295:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757452/real 1787757452] req@ffff8c05b2d99880 x1874588654927488/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757452.375474:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654927488/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757452.375485:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757452.375489:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757452.375493:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654927488:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757452.375501:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654927616:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757452.375527:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757452.375542:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757452/real 1787757452] req@ffff8c05b2d98380 x1874588654927616/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757452.375556:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654927616/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757452.375565:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757452.375569:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757452.375572:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654927616:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757452.375579:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654927744:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757452.375598:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757452.375609:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757452/real 1787757452] req@ffff8c05b2d9b480 x1874588654927744/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757452.375623:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654927744/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757452.375641:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757452.375645:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757452.375649:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654927744:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757452.375655:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654927872:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757452.375674:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757452.375685:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757452/real 1787757452] req@ffff8c05b2d98000 x1874588654927872/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757452.375698:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654927872/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757452.375707:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757452.375711:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757452.375714:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654927872:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757452.375720:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654928000:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757452.375738:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757452.375748:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757452/real 1787757452] req@ffff8c05b2d99f80 x1874588654928000/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757452.375762:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654928000/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757452.375771:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757452.375775:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757452.375780:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654928000:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757452.502019:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757452.950213:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757453.526000:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757453.974133:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757454.549952:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757454.998367:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757455.574030:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757456.022191:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757456.598081:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757457.046197:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757457.494028:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757457.494039:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787757457.494044:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757457.494055:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10513 00000100:00080000:1.0:1787757457.494066:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757457.494074:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757457.494112:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588654928128/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757457.494138:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757457.494142:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757457.494144:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757457.494148:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10513 00000100:00080000:1.0:1787757457.494151:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757457.494154:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757457.494163:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757457.494176:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588654928256/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757457.494187:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757457.494191:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1787757457.494199:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654928128:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757457.494205:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757457.494213:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10513 00000100:00080000:1.0:1787757457.494216:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757457.494239:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757457.494245:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757457.494261:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654928384/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757457.494272:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757457.494276:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757457.494278:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757457.494282:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10513 00000100:00080000:1.0:1787757457.494288:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757457.494291:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757457.494297:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787757457.494300:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757457.494312:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654928512/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757457.494323:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757457.494326:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757457.494328:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757457.494331:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10513 00000100:00080000:1.0:1787757457.494336:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757457.494340:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757457.494345:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757457.494350:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757457/real 1787757457] req@ffff8c05b2d9a300 x1874588654928128/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757457.494361:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588654928640/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757457.494374:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757457.494380:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1787757457.494381:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588654928128/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757457.494383:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757457.494394:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10513 00000100:00080000:1.0:1787757457.494402:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757457.494408:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:3.0:1787757457.494410:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787757457.494413:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787757457.494419:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757457.494427:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654928768/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757457.494429:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654928128:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757457.494454:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757457.494457:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787757457.494459:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757457.494464:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10513 00000100:00100000:3.0:1787757457.494467:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654928256:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787757457.494470:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757457.494499:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757457.494506:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757457.494524:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654928896/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787757457.494554:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757457.494558:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757457.494563:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757457.494566:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757457.494571:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10513 00000100:00100000:3.0:1787757457.494577:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757457/real 1787757457] req@ffff8c05b2d98700 x1874588654928256/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757457.494578:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757457.494589:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757457.494598:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757457.494612:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654928256/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757457.494616:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588654929024/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757457.494659:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757457.494669:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757457.494677:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654928256:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757457.494686:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654928384:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757457.494715:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757457.494727:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757457/real 1787757457] req@ffff8c05b2d99f80 x1874588654928384/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757457.494739:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654928384/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757457.494747:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757457.494750:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757457.494753:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654928384:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757457.494758:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654928512:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757457.494774:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757457.494783:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757457/real 1787757457] req@ffff8c05b2d98000 x1874588654928512/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757457.494793:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654928512/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757457.494800:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757457.494803:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757457.494806:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654928512:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757457.494810:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654928640:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757457.494824:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757457.494833:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757457/real 1787757457] req@ffff8c05b2d9b480 x1874588654928640/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757457.494877:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654928640/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757457.494893:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757457.494899:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757457.494908:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654928640:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757457.494991:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654928768:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757457.495019:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757457.495032:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757457/real 1787757457] req@ffff8c05b2d98380 x1874588654928768/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757457.495044:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654928768/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757457.495052:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757457.495055:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757457.495058:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654928768:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757457.495077:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654928896:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757457.495094:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757457.495103:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757457/real 1787757457] req@ffff8c05b2d99880 x1874588654928896/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757457.495113:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654928896/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757457.495127:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757457.495130:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757457.495133:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654928896:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757457.495138:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654929024:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757457.495153:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757457.495160:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757457/real 1787757457] req@ffff8c05b2d9b100 x1874588654929024/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757457.495171:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654929024/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757457.495178:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757457.495181:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757457.495183:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654929024:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757457.622022:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757458.069986:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757458.646013:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757459.094156:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757459.670009:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757460.118127:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757460.694010:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757461.142285:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757461.718018:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757462.166159:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757462.614091:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757462.614104:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757462.614109:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757462.614122:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10519 00000100:00080000:3.0:1787757462.614160:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757462.614174:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757462.614234:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588654929152/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757462.614293:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757462.614298:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757462.614301:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757462.614308:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10519 00000100:00080000:3.0:1787757462.614314:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757462.614319:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757462.614332:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757462.614353:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588654929280/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757462.614368:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757462.614374:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757462.614378:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757462.614383:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10519 00000100:00080000:3.0:1787757462.614387:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757462.614395:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757462.614405:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757462.614406:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654929152:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757462.614424:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588654929408/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757462.614449:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757462.614456:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757462.614460:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757462.614469:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10519 00000100:00080000:3.0:1787757462.614478:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757462.614489:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757462.614505:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757462.614533:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588654929536/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757462.614550:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757462.614588:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757462.614595:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757462.614599:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757462.614607:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10519 00000100:00080000:3.0:1787757462.614614:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757462.614623:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757462.614635:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757462.614646:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757462/real 1787757462] req@ffff8c05a9f7ad80 x1874588654929152/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757462.614657:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588654929664/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757462.614729:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757462.614733:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757462.614736:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757462.614742:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588654929152/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757462.614750:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10519 00000100:00080000:3.0:1787757462.614767:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757462.614775:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757462.614782:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757462.614799:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588654929792/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757462.614801:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757462.614816:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757462.614833:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654929152:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787757462.614859:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654929280:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757462.614974:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757462.614983:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757462.614988:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787757462.615001:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757462.615011:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10519 00000100:00080000:3.0:1787757462.615023:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757462.615029:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757462.615041:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757462.615047:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757462/real 1787757462] req@ffff8c05a9f7bb80 x1874588654929280/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757462.615068:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588654929920/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757462.615075:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588654929280/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757462.615159:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757462.615170:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787757462.615174:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787757462.615179:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654929280:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757462.615204:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757462.615211:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757462.615222:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10519 00000100:00080000:3.0:1787757462.615232:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757462.615249:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654929408:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757462.615253:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757462.615267:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787757462.615306:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757462.615306:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588654930048/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757462.615342:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757462/real 1787757462] req@ffff8c05a9f79180 x1874588654929408/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757462.615364:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588654929408/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757462.615387:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757462.615393:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757462.615397:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654929408:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757462.615407:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654929536:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757462.615431:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757462.615444:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757462/real 1787757462] req@ffff8c05a9f79f80 x1874588654929536/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757462.615458:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588654929536/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757462.615469:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757462.615473:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757462.615476:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654929536:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757462.615484:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654929664:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757462.615505:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757462.615516:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757462/real 1787757462] req@ffff8c05a9f78e00 x1874588654929664/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757462.615531:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588654929664/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757462.615540:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757462.615544:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757462.615549:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654929664:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757462.615556:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654929792:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757462.615576:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757462.615587:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757462/real 1787757462] req@ffff8c05a9f79500 x1874588654929792/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757462.615601:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588654929792/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757462.615611:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757462.615615:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757462.615618:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654929792:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757462.615625:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654929920:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757462.615644:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757462.615655:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757462/real 1787757462] req@ffff8c05a9f7b100 x1874588654929920/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757462.615669:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588654929920/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757462.615679:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757462.615683:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757462.615686:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654929920:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757462.615709:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654930048:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757462.615730:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757462.615743:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757462/real 1787757462] req@ffff8c05a9f7b800 x1874588654930048/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757462.615757:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588654930048/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757462.615767:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757462.615771:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757462.615774:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654930048:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757462.742009:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757463.190254:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757463.766049:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757464.214126:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757464.790015:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757465.238246:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757465.814146:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757466.262213:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757466.837978:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757467.286257:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757467.734023:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757467.734036:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757467.734069:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757467.734082:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10524 00000100:00080000:3.0:1787757467.734095:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757467.734105:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757467.734151:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588654930176/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757467.734187:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757467.734192:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757467.734195:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757467.734200:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10524 00000100:00080000:3.0:1787757467.734204:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757467.734209:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757467.734221:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757467.734237:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588654930304/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757467.734253:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787757467.734262:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654930176:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757467.734263:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757467.734270:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757467.734284:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10524 00000100:00080000:3.0:1787757467.734289:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757467.734293:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757467.734300:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757467.734316:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588654930432/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757467.734337:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757467.734342:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757467.734344:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757467.734349:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10524 00000100:00080000:3.0:1787757467.734354:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757467.734358:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757467.734365:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757467.734377:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588654930560/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757467.734389:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757467.734393:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757467.734396:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757467.734401:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10524 00000100:00080000:3.0:1787757467.734405:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757467.734410:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757467.734419:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757467.734445:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654930304:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757467.734446:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588654930688/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757467.734484:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757467.734491:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757467.734496:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757467.734506:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10524 00000100:00080000:3.0:1787757467.734517:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757467.734525:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654930432:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757467.734535:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757467.734550:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757467.734581:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654930560:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757467.734585:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588654930816/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757467.734614:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757467.734621:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757467.734626:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757467.734635:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10524 00000100:00080000:3.0:1787757467.734644:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757467.734649:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654930688:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757467.734659:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757467.734676:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757467.734705:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588654930944/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757467.734726:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787757467.734736:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654930816:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757467.734737:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757467.734747:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757467.734760:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10524 00000100:00080000:3.0:1787757467.734769:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757467.734778:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757467.734790:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757467.734804:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654930944:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757467.734821:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588654931072/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757467.735036:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654931072:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757467.862012:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757468.310115:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757468.885988:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757469.334166:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757469.910047:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757470.358119:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787757470.806135:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.107:988 00000400:00000100:1.0:1787757470.806171:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.107@tcp at host 192.168.201.107 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787757470.806184:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787757470.806216:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787757470.806223:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787757470.806229:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787757470.806236:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787757470.806242:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787757470.806248:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787757470.806255:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:0.0:1787757470.806270:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757467/real 1787757470] req@ffff8c05a9f7bb80 x1874588654930944/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757470.806297:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588654930944/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757470.806309:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757470.806315:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757470.806319:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654930944:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757470.806327:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757467/real 1787757470] req@ffff8c05a9f79f80 x1874588654930688/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757470.806339:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588654930688/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757470.806346:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757470.806350:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757470.806354:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654930688:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757470.806360:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757467/real 1787757470] req@ffff8c05a9f79180 x1874588654930816/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757470.806372:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588654930816/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757470.806379:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757470.806384:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757470.806393:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654930816:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757470.806399:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757467/real 1787757470] req@ffff8c05a9f7b100 x1874588654930304/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757470.806411:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588654930304/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757470.806420:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757470.806423:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757470.806427:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654930304:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757470.806433:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757467/real 1787757470] req@ffff8c05a9f79500 x1874588654930432/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757470.806445:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588654930432/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757470.806452:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757470.806456:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757470.806459:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654930432:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757470.806466:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757467/real 1787757470] req@ffff8c05a9f78e00 x1874588654930560/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757470.806476:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588654930560/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757470.806490:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757470.806494:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757470.806497:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654930560:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757470.806504:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757467/real 1787757470] req@ffff8c05a9f7b800 x1874588654930176/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757470.806517:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588654930176/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757470.806525:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757470.806528:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757470.806532:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654930176:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787757470.806562:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757467/real 1787757470] req@ffff8c05a9f7ad80 x1874588654931072/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757470.806574:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588654931072/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757470.806581:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757470.806585:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757470.806588:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654931072:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757470.933935:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757471.382230:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757471.958063:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757472.406161:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757472.854114:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757472.854130:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787757472.854137:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757472.854154:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10529 00000100:00080000:2.0:1787757472.854169:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757472.854182:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757472.854239:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588654931200/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757472.854292:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757472.854297:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757472.854300:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757472.854306:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10529 00000100:00080000:2.0:1787757472.854312:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757472.854318:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757472.854331:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757472.854352:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588654931328/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757472.854369:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757472.854373:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757472.854376:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757472.854381:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10529 00000100:00080000:2.0:1787757472.854385:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757472.854395:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757472.854401:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654931200:192.168.201.107@tcp:250:kworker.0 00000008:00100000:2.0:1787757472.854405:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757472.854433:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588654931456/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757472.854456:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757472.854464:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757472.854467:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757472.854476:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10529 00000100:00080000:2.0:1787757472.854483:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757472.854491:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757472.854503:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757472.854528:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588654931584/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757472.854548:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757472.854555:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757472.854559:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757472.854566:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10529 00000100:00080000:2.0:1787757472.854573:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757472.854581:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757472.854591:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787757472.854596:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757472.854630:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588654931712/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757472.854653:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757472.854659:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757472.854664:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757472.854674:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10529 00000100:00080000:2.0:1787757472.854686:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757472.854694:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757472.854706:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757472.854714:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757472/real 1787757472] req@ffff8c05b2e73100 x1874588654931200/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757472.854740:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588654931840/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757472.854807:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757472.854814:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787757472.854822:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588654931200/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757472.854823:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757472.854851:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10529 00000100:00080000:2.0:1787757472.854869:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757472.854980:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757472.854997:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757472.855010:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654931200:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757472.855035:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757472.855051:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757472.855087:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588654931968/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757472.855094:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654931328:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757472.855148:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757472.855156:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757472.855161:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757472.855175:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10529 00000100:00080000:2.0:1787757472.855185:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787757472.855198:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757472.855208:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757472.855224:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757472.855250:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757472/real 1787757472] req@ffff8c05b2e70700 x1874588654931328/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757472.855264:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588654932096/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757472.855318:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588654931328/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757472.855341:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757472.855363:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757472.855373:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654931328:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757472.855390:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654931456:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757472.855471:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757472.855499:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757472/real 1787757472] req@ffff8c05b2e73480 x1874588654931456/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757472.855527:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588654931456/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757472.855555:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757472.855562:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757472.855569:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654931456:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757472.855582:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654931584:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757472.855627:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757472.855651:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757472/real 1787757472] req@ffff8c05b2e71500 x1874588654931584/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757472.855676:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588654931584/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757472.855693:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757472.855703:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757472.855711:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654931584:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757472.855729:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654931712:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757472.855774:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757472.855815:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757472/real 1787757472] req@ffff8c05b2e73b80 x1874588654931712/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757472.855842:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588654931712/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757472.855858:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757472.855865:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757472.855872:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654931712:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757472.855941:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654931840:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757472.856011:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757472.856039:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757472/real 1787757472] req@ffff8c05b2e71c00 x1874588654931840/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757472.856065:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588654931840/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757472.856083:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757472.856089:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757472.856096:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654931840:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757472.856145:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654931968:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757472.856199:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757472.856225:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757472/real 1787757472] req@ffff8c05b2e71880 x1874588654931968/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757472.856250:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588654931968/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757472.856268:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757472.856275:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757472.856282:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654931968:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757472.856295:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654932096:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757472.856336:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757472.856361:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757472/real 1787757472] req@ffff8c05b2e70380 x1874588654932096/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757472.856382:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588654932096/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757472.856394:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757472.856398:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757472.856402:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654932096:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757472.982051:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757473.430351:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757474.006093:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757474.454186:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757475.030053:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757475.478184:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757476.053978:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757476.502080:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757477.078028:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757477.526189:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757477.974017:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757477.974029:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787757477.974034:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757477.974045:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10534 00000100:00080000:2.0:1787757477.974057:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757477.974065:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757477.974110:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73800 x1874588654932224/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757477.974143:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757477.974147:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757477.974150:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757477.974154:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10534 00000100:00080000:2.0:1787757477.974158:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757477.974161:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757477.974171:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757477.974187:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588654932352/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757477.974199:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757477.974204:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757477.974206:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757477.974210:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10534 00000100:00080000:2.0:1787757477.974214:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757477.974222:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654932224:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757477.974246:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757477.974262:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757477.974281:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588654932480/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757477.974302:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757477.974309:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757477.974312:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757477.974319:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10534 00000100:00080000:2.0:1787757477.974327:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757477.974336:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757477.974345:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787757477.974353:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757477.974375:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588654932608/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757477.974395:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757477.974399:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757477.974402:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757477.974408:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10534 00000100:00080000:2.0:1787757477.974414:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757477.974419:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757477.974425:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757477.974443:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757477/real 1787757477] req@ffff8c05b2e73800 x1874588654932224/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757477.974445:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588654932736/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757477.974499:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757477.974504:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757477.974506:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757477.974513:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73800 x1874588654932224/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757477.974520:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10534 00000100:00080000:2.0:1787757477.974530:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757477.974542:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787757477.974547:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787757477.974549:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787757477.974560:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757477.974572:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654932224:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787757477.974572:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588654932864/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757477.974596:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654932352:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757477.974601:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757477.974608:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787757477.974611:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757477.974618:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10534 00000100:00080000:2.0:1787757477.974623:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757477.974627:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757477.974632:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757477.974647:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588654932992/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757477.974658:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757477.974672:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757477.974677:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757477.974679:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757477.974705:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757477/real 1787757477] req@ffff8c05b2e70380 x1874588654932352/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757477.974706:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10534 00000100:00080000:2.0:1787757477.974717:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757477.974730:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757477.974742:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757477.974747:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588654932352/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757477.974779:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787757477.974780:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588654933120/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757477.974790:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757477.974803:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654932352:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757477.974845:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654932480:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757477.974959:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757477.974976:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757477/real 1787757477] req@ffff8c05b2e71880 x1874588654932480/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757477.974990:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588654932480/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757477.974999:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757477.975003:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757477.975007:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654932480:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757477.975013:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654932608:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757477.975033:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757477.975043:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757477/real 1787757477] req@ffff8c05b2e71c00 x1874588654932608/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757477.975055:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588654932608/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757477.975064:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757477.975068:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757477.975071:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654932608:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757477.975077:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654932736:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757477.975093:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757477.975104:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757477/real 1787757477] req@ffff8c05b2e73b80 x1874588654932736/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757477.975123:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588654932736/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757477.975136:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757477.975142:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757477.975148:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654932736:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757477.975158:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654932864:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757477.975193:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757477.975212:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757477/real 1787757477] req@ffff8c05b2e71500 x1874588654932864/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757477.975243:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588654932864/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757477.975253:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757477.975257:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757477.975262:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654932864:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757477.975269:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654932992:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757477.975295:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757477.975315:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757477/real 1787757477] req@ffff8c05b2e73480 x1874588654932992/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757477.975336:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588654932992/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757477.975352:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757477.975359:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757477.975366:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654932992:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757477.975378:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654933120:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757477.975409:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757477.975424:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757477/real 1787757477] req@ffff8c05b2e70700 x1874588654933120/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757477.975438:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588654933120/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757477.975448:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757477.975452:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757477.975456:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654933120:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757478.101985:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757478.550038:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757479.125982:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757479.574108:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757480.150023:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757480.598062:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757481.173989:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757481.622296:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757482.197979:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757482.646066:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757483.094008:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757483.094016:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757483.094020:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757483.094028:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10539 00000100:00080000:3.0:1787757483.094037:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757483.094042:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757483.094074:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588654933248/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757483.094098:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757483.094101:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757483.094102:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757483.094105:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10539 00000100:00080000:3.0:1787757483.094107:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757483.094110:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757483.094119:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757483.094132:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588654933376/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757483.094143:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757483.094147:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757483.094149:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757483.094155:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10539 00000100:00100000:0.0:1787757483.094159:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654933248:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757483.094161:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757483.094169:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757483.094175:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757483.094188:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588654933504/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757483.094198:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757483.094200:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757483.094202:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757483.094205:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10539 00000100:00080000:3.0:1787757483.094208:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757483.094210:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757483.094214:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757483.094222:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588654933632/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757483.094229:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757483.094233:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:0.0:1787757483.094235:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757483.094236:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757483.094242:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10539 00000100:00080000:3.0:1787757483.094244:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757483.094247:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757483.094251:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757483.094260:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588654933760/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757483.094266:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757483.094268:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757483.094269:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757483.094273:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10539 00000100:00080000:3.0:1787757483.094275:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757483.094278:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757483.094283:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757483/real 1787757483] req@ffff8c05a9f79880 x1874588654933248/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787757483.094283:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757483.094294:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588654933888/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757483.094305:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757483.094307:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757483.094309:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757483.094315:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10539 00000100:00100000:0.0:1787757483.094316:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588654933248/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757483.094317:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757483.094323:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757483.094329:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787757483.094340:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787757483.094340:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588654934016/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757483.094351:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787757483.094351:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757483.094357:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757483.094359:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757483.094362:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10539 00000100:00080000:3.0:1787757483.094363:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757483.094366:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757483.094371:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654933248:192.168.201.107@tcp:250:kworker.0 00000008:00100000:3.0:1787757483.094371:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757483.094381:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588654934144/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757483.094387:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654933376:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757483.094421:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757483.094436:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757483/real 1787757483] req@ffff8c05a9f7a300 x1874588654933376/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757483.094451:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588654933376/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757483.094460:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757483.094464:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757483.094467:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654933376:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757483.094492:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654933504:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757483.094514:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757483.094524:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757483/real 1787757483] req@ffff8c05a9f7a680 x1874588654933504/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757483.094536:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588654933504/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757483.094542:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757483.094544:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757483.094546:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654933504:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757483.094550:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654933632:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757483.094562:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757483.094569:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757483/real 1787757483] req@ffff8c05a9f78a80 x1874588654933632/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757483.094576:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588654933632/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757483.094581:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757483.094583:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757483.094585:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654933632:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757483.094588:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654933760:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757483.094598:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757483.094604:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757483/real 1787757483] req@ffff8c05a9f78380 x1874588654933760/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757483.094611:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588654933760/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757483.094620:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757483.094621:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757483.094623:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654933760:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757483.094627:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654933888:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757483.094636:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757483.094641:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757483/real 1787757483] req@ffff8c05a9f7b480 x1874588654933888/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757483.094648:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588654933888/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757483.094653:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757483.094655:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757483.094657:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654933888:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757483.094660:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654934016:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757483.094669:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757483.094674:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757483/real 1787757483] req@ffff8c05a9f78700 x1874588654934016/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757483.094681:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588654934016/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757483.094686:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757483.094687:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757483.094690:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654934016:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757483.094693:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654934144:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757483.094703:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757483.094708:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757483/real 1787757483] req@ffff8c05a9f7aa00 x1874588654934144/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757483.094715:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588654934144/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757483.094720:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757483.094722:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757483.094724:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654934144:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757483.221943:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757483.670086:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757484.245940:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757484.694097:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757485.269947:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757485.718232:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757486.294049:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757486.742130:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757487.318023:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757487.766211:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757488.214051:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757488.214063:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787757488.214070:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757488.214084:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10544 00000100:00080000:0.0:1787757488.214098:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757488.214110:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757488.214159:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588654934272/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757488.214198:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757488.214204:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757488.214207:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757488.214215:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10544 00000100:00080000:0.0:1787757488.214221:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757488.214227:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757488.214241:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757488.214262:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588654934400/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757488.214270:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654934272:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757488.214297:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757488.214303:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757488.214306:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757488.214313:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10544 00000100:00080000:0.0:1787757488.214318:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757488.214324:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757488.214331:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757488.214349:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588654934528/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757488.214364:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757488.214369:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757488.214371:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757488.214375:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10544 00000100:00080000:0.0:1787757488.214379:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757488.214388:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:3.0:1787757488.214392:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:0.0:1787757488.214396:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757488.214412:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588654934656/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757488.214425:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757488.214428:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757488.214430:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757488.214434:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10544 00000100:00080000:0.0:1787757488.214437:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757488.214442:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757488.214463:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757488.214474:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588654934784/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757488.214485:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757488.214489:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757488.214491:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757488.214495:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10544 00000100:00080000:0.0:1787757488.214499:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757488.214504:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757488.214510:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757488.214525:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588654934912/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757488.214538:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757488.214543:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787757488.214545:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757488.214551:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10544 00000100:00100000:3.0:1787757488.214560:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757488/real 1787757488] req@ffff8c05a9f76300 x1874588654934272/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757488.214561:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757488.214573:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757488.214578:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757488.214591:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588654935040/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757488.214601:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757488.214605:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757488.214607:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757488.214611:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10544 00000100:00080000:0.0:1787757488.214615:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757488.214619:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757488.214624:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757488.214634:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588654935168/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757488.214830:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588654934272/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757488.214958:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757488.214967:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757488.214975:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654934272:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757488.215015:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654934400:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757488.215059:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757488.215079:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757488/real 1787757488] req@ffff8c05a9f75180 x1874588654934400/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757488.215096:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588654934400/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757488.215113:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757488.215120:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757488.215127:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654934400:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757488.215140:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654934528:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757488.215181:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757488.215206:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757488/real 1787757488] req@ffff8c05a9f77100 x1874588654934528/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757488.215229:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588654934528/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757488.215240:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757488.215244:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757488.215249:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654934528:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757488.215257:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654934656:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757488.215287:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757488.215305:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757488/real 1787757488] req@ffff8c05a9f75500 x1874588654934656/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757488.215328:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588654934656/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757488.215345:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757488.215352:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757488.215360:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654934656:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757488.215372:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654934784:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757488.215414:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757488.215432:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757488/real 1787757488] req@ffff8c05a9f75880 x1874588654934784/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757488.215447:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588654934784/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757488.215458:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757488.215462:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757488.215466:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654934784:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757488.215473:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654934912:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757488.215495:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757488.215507:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757488/real 1787757488] req@ffff8c05a9f74000 x1874588654934912/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757488.215530:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588654934912/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757488.215540:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757488.215544:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757488.215547:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654934912:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757488.215554:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654935040:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757488.215576:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757488.215594:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757488/real 1787757488] req@ffff8c05a9f74e00 x1874588654935040/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757488.215618:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588654935040/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757488.215635:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757488.215641:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757488.215648:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654935040:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757488.215661:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654935168:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757488.215731:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757488.215756:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757488/real 1787757488] req@ffff8c05a9f76a00 x1874588654935168/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757488.215782:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588654935168/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757488.215806:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757488.215811:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757488.215815:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654935168:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757488.342006:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757488.790124:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757489.365985:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757489.814114:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757490.390046:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757490.838111:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757491.414004:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757491.862268:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757492.438068:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757492.886095:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757493.334033:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757493.334051:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787757493.334058:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757493.334075:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10549 00000100:00080000:0.0:1787757493.334092:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757493.334104:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757493.334163:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588654935296/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757493.334204:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757493.334210:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757493.334213:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757493.334218:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10549 00000100:00080000:0.0:1787757493.334224:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757493.334229:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757493.334241:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757493.334260:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588654935424/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757493.334276:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757493.334281:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757493.334285:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757493.334294:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654935296:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757493.334295:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10549 00000100:00080000:0.0:1787757493.334303:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757493.334314:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757493.334321:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757493.334337:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588654935552/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757493.334351:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757493.334358:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757493.334362:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757493.334370:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10549 00000100:00080000:0.0:1787757493.334377:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757493.334386:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757493.334397:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787757493.334408:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757493.334428:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588654935680/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757493.334452:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757493.334460:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757493.334463:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757493.334468:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757493/real 1787757493] req@ffff8c05a9f76d80 x1874588654935296/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757493.334475:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10549 00000100:00080000:0.0:1787757493.334486:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757493.334499:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757493.334509:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757493.334512:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588654935296/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757493.334533:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787757493.334534:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588654935808/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757493.334549:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787757493.334566:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787757493.334566:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654935296:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757493.334578:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757493.334584:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757493.334616:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10549 00000100:00080000:0.0:1787757493.334622:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757493.334628:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757493.334635:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757493.334650:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654935424:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757493.334656:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588654935936/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757493.334676:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757493.334681:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787757493.334684:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757493.334690:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10549 00000100:00080000:0.0:1787757493.334696:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757493.334701:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757493.334708:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787757493.334722:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757493.334727:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588654936064/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757493.334742:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757493.334747:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757493.334750:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757493.334755:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757493/real 1787757493] req@ffff8c05a9f77b80 x1874588654935424/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757493.334760:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10549 00000100:00080000:0.0:1787757493.334769:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757493.334773:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757493.334779:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757493.334792:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588654936192/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757493.334838:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588654935424/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757493.334854:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757493.334859:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757493.334867:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654935424:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757493.334948:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654935552:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757493.335003:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757493.335025:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757493/real 1787757493] req@ffff8c05a9f76680 x1874588654935552/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757493.335042:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588654935552/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757493.335053:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757493.335058:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757493.335062:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654935552:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757493.335070:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654935680:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757493.335097:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757493.335109:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757493/real 1787757493] req@ffff8c05a9f75c00 x1874588654935680/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757493.335124:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588654935680/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757493.335143:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757493.335147:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757493.335151:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654935680:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757493.335158:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654935808:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757493.335177:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757493.335189:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757493/real 1787757493] req@ffff8c05a9f77800 x1874588654935808/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757493.335203:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588654935808/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757493.335213:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757493.335217:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757493.335220:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654935808:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757493.335244:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654935936:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757493.335275:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757493.335296:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757493/real 1787757493] req@ffff8c05a9f74380 x1874588654935936/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757493.335323:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588654935936/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757493.335340:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757493.335347:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757493.335355:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654935936:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757493.335368:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654936064:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757493.335399:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757493.335414:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757493/real 1787757493] req@ffff8c05a9f74a80 x1874588654936064/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757493.335429:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588654936064/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757493.335440:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757493.335444:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757493.335448:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654936064:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757493.335455:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654936192:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757493.335474:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757493.335485:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757493/real 1787757493] req@ffff8c05a9f76d80 x1874588654936192/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757493.335499:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588654936192/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757493.335509:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757493.335512:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757493.335516:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654936192:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757493.462002:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757493.910250:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757494.486139:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757494.934134:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757495.509992:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757495.958247:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757496.534086:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757496.982215:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757497.558061:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757498.006224:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757498.454018:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757498.454029:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787757498.454034:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757498.454047:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10554 00000100:00080000:0.0:1787757498.454060:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757498.454068:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757498.454112:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588654936320/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757498.454138:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757498.454142:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757498.454145:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757498.454149:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10554 00000100:00080000:0.0:1787757498.454153:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757498.454158:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757498.454169:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757498.454187:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588654936448/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757498.454203:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757498.454206:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757498.454208:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757498.454210:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654936320:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757498.454216:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10554 00000100:00080000:0.0:1787757498.454221:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757498.454228:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757498.454233:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757498.454245:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588654936576/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757498.454258:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757498.454261:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757498.454262:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757498.454266:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10554 00000100:00080000:0.0:1787757498.454269:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757498.454273:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757498.454276:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757498.454285:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588654936704/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757498.454291:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757498.454294:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757498.454300:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787757498.454301:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757498.454313:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10554 00000100:00080000:0.0:1787757498.454316:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757498.454319:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757498.454324:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757498.454333:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588654936832/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757498.454341:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757498.454344:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757498.454345:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757498.454347:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10554 00000100:00080000:0.0:1787757498.454351:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757498.454354:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757498.454359:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757498.454367:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588654936960/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757498.454372:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757498.454374:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787757498.454376:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757498.454384:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757498/real 1787757498] req@ffff8c05a9f76a00 x1874588654936320/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757498.454387:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10554 00000100:00080000:0.0:1787757498.454396:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757498.454408:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757498.454416:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757498.454427:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588654937088/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757498.454436:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588654936320/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757498.454437:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757498.454450:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757498.454452:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757498.454465:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10554 00000100:00080000:3.0:1787757498.454466:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757498.454469:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757498.454476:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:3.0:1787757498.454479:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000008:00100000:0.0:1787757498.454482:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757498.454491:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654936320:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787757498.454494:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588654937216/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757498.454545:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654936448:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757498.454575:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757498.454588:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757498/real 1787757498] req@ffff8c05a9f74e00 x1874588654936448/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757498.454601:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588654936448/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757498.454611:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757498.454615:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757498.454619:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654936448:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757498.454626:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654936576:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757498.454649:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757498.454658:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757498/real 1787757498] req@ffff8c05a9f74000 x1874588654936576/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757498.454671:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588654936576/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757498.454677:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757498.454679:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757498.454682:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654936576:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757498.454687:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654936704:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757498.454700:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757498.454706:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757498/real 1787757498] req@ffff8c05a9f75880 x1874588654936704/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757498.454714:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588654936704/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757498.454720:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757498.454721:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757498.454723:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654936704:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757498.454727:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654936832:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757498.454739:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757498.454749:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757498/real 1787757498] req@ffff8c05a9f75500 x1874588654936832/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757498.454761:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588654936832/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757498.454777:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757498.454781:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757498.454784:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654936832:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757498.454790:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654936960:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757498.454815:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757498.454822:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757498/real 1787757498] req@ffff8c05a9f77100 x1874588654936960/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757498.454830:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588654936960/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757498.454835:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757498.454838:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757498.454840:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654936960:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757498.454844:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654937088:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757498.454854:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757498.454860:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757498/real 1787757498] req@ffff8c05a9f75180 x1874588654937088/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757498.454867:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588654937088/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757498.454873:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757498.454874:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757498.454910:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654937088:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757498.454915:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654937216:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757498.454929:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757498.454936:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757498/real 1787757498] req@ffff8c05a9f76300 x1874588654937216/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757498.454949:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588654937216/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757498.454959:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757498.454963:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757498.454966:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654937216:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757498.581992:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757499.029995:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757499.605990:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757500.054228:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757500.630028:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757501.078093:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757501.654067:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757502.102247:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757502.678084:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757503.126172:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757503.574075:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757503.574088:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787757503.574094:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757503.574106:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10559 00000100:00080000:1.0:1787757503.574120:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757503.574129:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757503.574175:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588654937344/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757503.574213:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757503.574218:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757503.574221:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757503.574228:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10559 00000100:00080000:1.0:1787757503.574236:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757503.574243:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757503.574259:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757503.574293:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654937344:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787757503.574295:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588654937472/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757503.574330:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757503.574337:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757503.574341:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757503.574347:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10559 00000100:00080000:1.0:1787757503.574354:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757503.574360:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757503.574368:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757503.574389:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588654937600/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757503.574405:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757503.574436:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757503.574438:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787757503.574450:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757503.574451:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10559 00000100:00080000:1.0:1787757503.574465:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757503.574471:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757503.574480:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757503.574498:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654937728/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757503.574512:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757503.574518:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757503.574522:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757503.574527:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757503/real 1787757503] req@ffff8c05b2d9bb80 x1874588654937344/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757503.574537:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10559 00000100:00080000:1.0:1787757503.574552:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757503.574566:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757503.574574:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588654937344/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787757503.574581:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787757503.574596:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757503.574603:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757503.574610:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654937344:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787757503.574621:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654937856/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757503.574647:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654937472:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787757503.574650:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757503.574665:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757503.574670:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757503.574681:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10559 00000100:00080000:1.0:1787757503.574691:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787757503.574696:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757503.574708:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757503.574722:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757503/real 1787757503] req@ffff8c05b2d9aa00 x1874588654937472/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787757503.574724:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757503.574751:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588654937472/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757503.574765:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588654937984/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757503.574771:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757503.574788:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757503.574800:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654937472:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757503.574822:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654937600:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787757503.574829:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757503.574836:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787757503.574840:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757503.574849:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10559 00000100:00080000:1.0:1787757503.574857:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787757503.574861:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757503.574869:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757503.574937:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757503.574939:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757503/real 1787757503] req@ffff8c05b2d9b100 x1874588654937600/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757503.574966:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654938112/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757503.574971:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654937600/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757503.575007:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757503.575013:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757503.575015:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757503.575025:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757503.575028:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787757503.575036:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10559 00000100:00100000:3.0:1787757503.575043:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654937600:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787757503.575047:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757503.575062:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757503.575067:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654937728:192.168.201.107@tcp:8:kworker.0 00000008:00100000:1.0:1787757503.575073:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757503.575096:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654938240/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787757503.575112:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757503.575142:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757503/real 1787757503] req@ffff8c05b2d99880 x1874588654937728/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757503.575183:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654937728/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757503.575200:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757503.575207:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757503.575214:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654937728:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757503.575242:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654937856:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757503.575273:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757503.575288:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757503/real 1787757503] req@ffff8c05b2d98380 x1874588654937856/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757503.575303:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654937856/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757503.575314:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757503.575318:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757503.575322:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654937856:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757503.575329:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654937984:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757503.575347:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757503.575364:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757503/real 1787757503] req@ffff8c05b2d9b480 x1874588654937984/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757503.575387:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654937984/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757503.575438:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757503.575446:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757503.575454:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654937984:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757503.575466:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654938112:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757503.575501:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757503.575516:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757503/real 1787757503] req@ffff8c05b2d98000 x1874588654938112/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757503.575531:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654938112/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757503.575542:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757503.575546:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757503.575550:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654938112:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757503.575557:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654938240:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757503.575576:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757503.575587:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757503/real 1787757503] req@ffff8c05b2d99f80 x1874588654938240/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757503.575601:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654938240/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757503.575611:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757503.575614:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757503.575619:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654938240:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757503.702011:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757504.150204:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757504.726049:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757505.174104:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757505.750045:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757506.198257:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757506.773986:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757507.222036:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757507.797952:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757508.246210:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757508.694006:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757508.694019:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787757508.694026:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757508.694038:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10565 00000100:00080000:1.0:1787757508.694052:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757508.694061:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757508.694108:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588654938368/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757508.694148:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757508.694153:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757508.694155:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757508.694160:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10565 00000100:00080000:1.0:1787757508.694164:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757508.694169:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757508.694180:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757508.694197:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588654938496/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757508.694209:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757508.694214:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757508.694216:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757508.694220:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10565 00000100:00080000:1.0:1787757508.694223:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757508.694227:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757508.694232:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757508.694245:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654938624/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757508.694270:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757508.694280:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787757508.694280:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654938368:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757508.694288:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757508.694303:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10565 00000100:00080000:1.0:1787757508.694306:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757508.694311:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757508.694341:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757508.694358:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654938752/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757508.694372:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757508.694378:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757508.694380:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757508.694385:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10565 00000100:00080000:1.0:1787757508.694391:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757508.694396:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757508.694402:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757508.694418:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588654938880/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787757508.694428:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757508.694440:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757508.694446:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757508.694449:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757508.694457:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10565 00000100:00080000:1.0:1787757508.694462:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757508.694467:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757508.694473:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757508.694489:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654939008/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757508.694499:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757508/real 1787757508] req@ffff8c05b2d98700 x1874588654938368/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757508.694510:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757508.694525:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787757508.694532:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757508.694542:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10565 00000100:00080000:1.0:1787757508.694548:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787757508.694559:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654938368/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757508.694560:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757508.694571:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757508.694590:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654939136/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757508.694600:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757508.694604:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757508.694616:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757508.694619:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757508.694623:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10565 00000100:00080000:1.0:1787757508.694626:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757508.694630:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757508.694634:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787757508.694640:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757508.694651:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588654939264/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757508.694657:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654938368:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757508.694682:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654938496:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757508.694754:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757508.694775:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757508/real 1787757508] req@ffff8c05b2d9a300 x1874588654938496/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757508.694796:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588654938496/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757508.694808:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757508.694816:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757508.694821:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654938496:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757508.694851:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654938624:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757508.694914:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757508.694932:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757508/real 1787757508] req@ffff8c05b2d99f80 x1874588654938624/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757508.694948:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654938624/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757508.694958:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757508.694962:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757508.694967:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654938624:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757508.694974:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654938752:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757508.694999:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757508.695011:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757508/real 1787757508] req@ffff8c05b2d98000 x1874588654938752/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757508.695040:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654938752/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757508.695050:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757508.695054:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757508.695058:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654938752:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757508.695065:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654938880:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757508.695087:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757508.695099:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757508/real 1787757508] req@ffff8c05b2d9b480 x1874588654938880/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757508.695113:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654938880/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757508.695123:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757508.695126:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757508.695130:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654938880:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757508.695239:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654939008:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757508.695272:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757508.695287:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757508/real 1787757508] req@ffff8c05b2d98380 x1874588654939008/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757508.695302:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654939008/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757508.695338:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757508.695346:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757508.695353:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654939008:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757508.695365:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654939136:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757508.695407:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757508.695430:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757508/real 1787757508] req@ffff8c05b2d99880 x1874588654939136/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757508.695458:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654939136/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757508.695469:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757508.695473:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757508.695477:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654939136:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757508.695485:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654939264:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757508.695511:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757508.695524:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757508/real 1787757508] req@ffff8c05b2d9b100 x1874588654939264/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757508.695537:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654939264/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757508.695551:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757508.695557:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757508.695563:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654939264:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757508.821993:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757509.270081:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757509.846002:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757510.294177:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757510.870030:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757511.318064:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757511.894003:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757512.342301:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757512.918057:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757513.366084:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757513.814072:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757513.814089:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757513.814094:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757513.814107:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10570 00000100:00080000:3.0:1787757513.814120:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757513.814130:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757513.814180:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588654939392/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757513.814246:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757513.814254:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757513.814258:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757513.814267:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10570 00000100:00080000:3.0:1787757513.814275:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757513.814284:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757513.814299:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757513.814327:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588654939520/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757513.814347:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757513.814352:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757513.814355:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757513.814361:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10570 00000100:00080000:3.0:1787757513.814367:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757513.814372:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757513.814380:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757513.814387:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654939392:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757513.814400:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588654939648/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757513.814419:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757513.814424:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757513.814426:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757513.814431:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10570 00000100:00080000:3.0:1787757513.814437:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757513.814442:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757513.814449:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757513.814463:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588654939776/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757513.814475:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787757513.814484:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757513.814486:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757513.814494:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757513.814500:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10570 00000100:00080000:3.0:1787757513.814506:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757513.814514:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757513.814525:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757513.814553:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588654939904/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757513.814570:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757513/real 1787757513] req@ffff8c05a9f7aa00 x1874588654939392/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757513.814583:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757513.814601:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757513.814609:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757513.814620:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10570 00000100:00080000:3.0:1787757513.814627:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757513.814632:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757513.814641:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757513.814651:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588654939392/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757513.814671:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588654940032/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757513.814685:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757513.814702:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787757513.814703:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757513.814713:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757513.814722:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757513.814725:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654939392:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757513.814738:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10570 00000100:00100000:0.0:1787757513.814748:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654939520:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757513.814752:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757513.814764:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757513.814774:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757513.814796:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588654940160/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757513.814803:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757513.814822:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757513.814827:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757513.814830:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757513.814836:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757513/real 1787757513] req@ffff8c05a9f78700 x1874588654939520/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757513.814838:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10570 00000100:00080000:3.0:1787757513.814849:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757513.814863:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757513.814950:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757513.814968:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588654939520/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757513.814985:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588654940288/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757513.815002:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757513.815017:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757513.815031:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654939520:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757513.815090:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654939648:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757513.815125:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757513.815142:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757513/real 1787757513] req@ffff8c05a9f7b480 x1874588654939648/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757513.815163:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588654939648/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757513.815180:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757513.815187:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757513.815194:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654939648:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757513.815207:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654939776:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757513.815362:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757513.815381:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757513/real 1787757513] req@ffff8c05a9f78380 x1874588654939776/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757513.815397:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588654939776/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757513.815408:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757513.815412:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757513.815417:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654939776:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757513.815425:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654939904:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757513.815447:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757513.815459:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757513/real 1787757513] req@ffff8c05a9f78a80 x1874588654939904/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757513.815473:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588654939904/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757513.815483:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757513.815486:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757513.815490:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654939904:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757513.815497:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654940032:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757513.815515:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757513.815527:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757513/real 1787757513] req@ffff8c05a9f7a680 x1874588654940032/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757513.815541:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588654940032/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757513.815551:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757513.815554:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757513.815558:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654940032:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757513.815565:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654940160:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757513.815589:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757513.815610:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757513/real 1787757513] req@ffff8c05a9f7a300 x1874588654940160/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757513.815635:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588654940160/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757513.815652:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757513.815658:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757513.815666:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654940160:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757513.815678:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654940288:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757513.815712:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757513.815728:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757513/real 1787757513] req@ffff8c05a9f79880 x1874588654940288/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757513.815743:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588654940288/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757513.815754:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757513.815758:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757513.815762:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654940288:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757513.941967:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757514.390189:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757514.966061:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757515.414024:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757515.989949:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757516.438090:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757517.014044:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757517.462213:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757518.037946:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757518.486259:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757518.933958:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757518.933966:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757518.933970:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757518.933978:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10575 00000100:00080000:3.0:1787757518.933987:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757518.933993:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757518.934025:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588654940416/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757518.934050:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757518.934053:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757518.934054:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757518.934057:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10575 00000100:00080000:3.0:1787757518.934060:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757518.934062:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757518.934070:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757518.934078:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588654940544/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757518.934086:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757518.934089:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757518.934091:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757518.934095:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10575 00000100:00080000:3.0:1787757518.934099:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757518.934102:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757518.934105:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757518.934112:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588654940672/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757518.934119:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757518.934121:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757518.934122:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757518.934125:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10575 00000100:00080000:3.0:1787757518.934127:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757518.934130:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757518.934133:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757518.934164:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588654940800/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757518.934177:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654940416:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757518.934182:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757518.934201:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757518.934204:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757518.934210:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10575 00000100:00080000:3.0:1787757518.934214:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757518.934219:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757518.934227:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757518.934240:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588654940928/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757518.934248:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757518.934251:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757518.934253:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757518.934257:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10575 00000100:00080000:3.0:1787757518.934260:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757518.934264:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757518.934268:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757518.934279:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588654941056/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757518.934285:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757518.934288:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757518.934289:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757518.934291:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10575 00000100:00080000:3.0:1787757518.934294:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757518.934297:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757518.934301:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757518.934307:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588654941184/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757518.934320:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757518.934330:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757518.934335:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757518.934338:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757518.934346:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10575 00000100:00080000:3.0:1787757518.934351:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757518.934356:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757518.934362:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757518.934374:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588654941312/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757518.934411:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757518/real 1787757518] req@ffff8c05a9f7ad80 x1874588654940416/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757518.934440:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588654940416/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757518.934457:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757518.934463:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757518.934469:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654940416:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787757518.934480:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654940544:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757518.934510:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757518.934527:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757518/real 1787757518] req@ffff8c05a9f7b800 x1874588654940544/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757518.934545:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588654940544/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757518.934562:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757518.934569:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757518.934575:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654940544:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757518.934588:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654940672:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757518.934642:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757518.934666:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757518/real 1787757518] req@ffff8c05a9f78e00 x1874588654940672/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757518.934684:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588654940672/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757518.934695:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757518.934700:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757518.934704:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654940672:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757518.934766:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654940800:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757518.934806:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757518.934821:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757518/real 1787757518] req@ffff8c05a9f79500 x1874588654940800/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757518.934836:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588654940800/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757518.934846:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757518.934851:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757518.934855:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654940800:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757518.934862:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654940928:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757518.934937:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757518.934967:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757518/real 1787757518] req@ffff8c05a9f7b100 x1874588654940928/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757518.934981:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588654940928/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757518.934992:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757518.934997:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757518.935001:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654940928:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757518.935009:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654941056:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757518.935032:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757518.935045:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757518/real 1787757518] req@ffff8c05a9f79180 x1874588654941056/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757518.935060:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588654941056/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757518.935070:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757518.935074:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757518.935078:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654941056:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757518.935085:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654941184:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757518.935119:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757518.935166:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757518/real 1787757518] req@ffff8c05a9f79f80 x1874588654941184/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757518.935199:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588654941184/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757518.935216:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757518.935224:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757518.935230:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654941184:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757518.935243:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654941312:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757518.935274:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757518.935290:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757518/real 1787757518] req@ffff8c05a9f7bb80 x1874588654941312/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757518.935305:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588654941312/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757518.935324:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757518.935328:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757518.935332:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654941312:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757519.061944:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757519.510149:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757520.086053:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757520.534242:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757521.110004:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757521.558121:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757522.134061:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757522.582264:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757523.157984:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757523.606125:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757524.053944:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757524.053953:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757524.053958:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757524.053969:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10580 00000100:00080000:3.0:1787757524.053981:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757524.053990:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757524.054033:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588654941440/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757524.054087:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757524.054092:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757524.054094:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757524.054105:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10580 00000100:00080000:3.0:1787757524.054110:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757524.054115:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757524.054126:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757524.054143:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588654941568/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757524.054157:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654941440:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757524.054158:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757524.054169:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757524.054171:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757524.054177:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10580 00000100:00080000:3.0:1787757524.054182:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757524.054186:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757524.054193:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757524.054207:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588654941696/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757524.054219:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757524.054223:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1787757524.054227:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757524.054229:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757524.054236:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10580 00000100:00080000:3.0:1787757524.054242:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757524.054248:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757524.054254:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757524.054273:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588654941824/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757524.054274:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757524/real 1787757524] req@ffff8c05a9f7bb80 x1874588654941440/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757524.054314:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588654941440/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757524.054320:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757524.054327:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757524.054328:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757524.054333:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757524.054335:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757524.054345:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654941440:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757524.054345:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10580 00000100:00080000:3.0:1787757524.054356:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757524.054364:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757524.054372:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757524.054373:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654941568:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757524.054393:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588654941952/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757524.054398:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757524.054412:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757524/real 1787757524] req@ffff8c05a9f79f80 x1874588654941568/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757524.054413:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757524.054426:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757524.054432:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757524.054435:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588654941568/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757524.054440:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10580 00000100:00080000:0.0:1787757524.054451:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757524.054451:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757524.054458:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787757524.054463:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757524.054464:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654941568:192.168.201.107@tcp:38:kworker.0 00000008:00100000:3.0:1787757524.054474:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757524.054475:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654941696:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757524.054499:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588654942080/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757524.054501:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757524.054511:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757524/real 1787757524] req@ffff8c05a9f79180 x1874588654941696/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757524.054517:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757524.054524:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787757524.054527:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588654941696/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757524.054529:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757524.054540:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10580 00000100:00080000:0.0:1787757524.054541:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757524.054549:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787757524.054550:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757524.054553:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654941696:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757524.054559:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757524.054560:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654941824:192.168.201.107@tcp:8:kworker.0 00000008:00100000:3.0:1787757524.054572:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787757524.054580:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757524.054593:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757524/real 1787757524] req@ffff8c05a9f7b100 x1874588654941824/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757524.054593:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588654942208/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757524.054627:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588654941824/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757524.054631:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757524.054637:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757524.054639:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757524.054642:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787757524.054645:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757524.054646:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654941824:192.168.201.107@tcp:8:kworker.0 00000100:00080000:3.0:1787757524.054657:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10580 00000100:00080000:3.0:1787757524.054664:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757524.054668:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654941952:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757524.054674:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757524.054682:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787757524.054687:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757524.054696:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757524/real 1787757524] req@ffff8c05a9f79500 x1874588654941952/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757524.054707:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588654941952/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757524.054716:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588654942336/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757524.054719:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757524.054729:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757524.054738:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654941952:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757524.054748:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654942080:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757524.054776:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757524.054787:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757524/real 1787757524] req@ffff8c05a9f78e00 x1874588654942080/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757524.054796:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588654942080/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757524.054802:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757524.054806:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757524.054809:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654942080:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757524.054814:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654942208:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757524.054830:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757524.054840:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757524/real 1787757524] req@ffff8c05a9f7b800 x1874588654942208/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757524.054849:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588654942208/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757524.054856:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757524.054859:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757524.054862:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654942208:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757524.054873:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654942336:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757524.054929:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757524.054940:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757524/real 1787757524] req@ffff8c05a9f7ad80 x1874588654942336/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757524.054950:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588654942336/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757524.054959:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757524.054962:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757524.054966:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654942336:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757524.181960:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757524.630101:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757525.206104:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757525.654129:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757526.230048:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757526.678238:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757527.253987:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757527.702070:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757528.278010:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757528.726183:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757529.174025:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757529.174037:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757529.174043:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757529.174055:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10585 00000100:00080000:3.0:1787757529.174068:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757529.174076:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757529.174118:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588654942464/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757529.174150:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757529.174154:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757529.174156:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757529.174160:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10585 00000100:00080000:3.0:1787757529.174165:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757529.174169:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757529.174178:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757529.174186:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654942464:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757529.174193:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588654942592/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757529.174203:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757529.174206:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757529.174208:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757529.174211:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10585 00000100:00080000:3.0:1787757529.174237:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757529.174240:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757529.174244:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757529.174257:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588654942720/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757529.174268:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757529.174272:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757529.174274:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757529.174278:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10585 00000100:00080000:3.0:1787757529.174282:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757529.174285:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757529.174290:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787757529.174314:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757529.174330:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588654942848/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757529.174346:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757529.174350:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757529.174352:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757529.174357:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10585 00000100:00080000:3.0:1787757529.174361:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757529.174367:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757529.174387:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757529.174390:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757529/real 1787757529] req@ffff8c05a9f79880 x1874588654942464/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757529.174423:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588654942976/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757529.174434:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787757529.174439:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588654942464/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757529.174441:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757529.174445:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757529.174457:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10585 00000100:00080000:3.0:1787757529.174465:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757529.174466:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757529.174472:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787757529.174495:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000008:00100000:3.0:1787757529.174498:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757529.174502:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654942464:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757529.174513:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588654943104/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757529.174521:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757529.174523:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757529.174525:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757529.174528:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654942592:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757529.174532:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10585 00000100:00080000:3.0:1787757529.174536:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757529.174540:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757529.174543:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757529.174554:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588654943232/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757529.174560:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757529.174571:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757529.174574:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757529.174576:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757529.174578:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757529/real 1787757529] req@ffff8c05a9f7a300 x1874588654942592/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757529.174582:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10585 00000100:00080000:3.0:1787757529.174589:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757529.174596:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757529.174601:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757529.174603:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588654942592/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757529.174612:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787757529.174613:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588654943360/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757529.174621:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757529.174629:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654942592:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757529.174638:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654942720:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757529.174654:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757529.174679:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757529/real 1787757529] req@ffff8c05a9f7a680 x1874588654942720/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757529.174691:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588654942720/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757529.174699:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757529.174701:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757529.174705:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654942720:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757529.174710:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654942848:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757529.174727:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757529.174736:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757529/real 1787757529] req@ffff8c05a9f78a80 x1874588654942848/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757529.174762:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588654942848/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757529.174768:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757529.174770:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757529.174772:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654942848:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757529.174776:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654942976:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757529.174788:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757529.174795:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757529/real 1787757529] req@ffff8c05a9f78380 x1874588654942976/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757529.174803:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588654942976/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757529.174827:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757529.174830:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757529.174834:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654942976:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757529.174841:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654943104:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757529.174860:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757529.174872:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757529/real 1787757529] req@ffff8c05a9f7b480 x1874588654943104/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757529.174907:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588654943104/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757529.174912:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757529.174914:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757529.174916:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654943104:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757529.174930:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654943232:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757529.174942:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757529.174949:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757529/real 1787757529] req@ffff8c05a9f78700 x1874588654943232/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757529.174990:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588654943232/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757529.174997:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757529.174999:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757529.175002:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654943232:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757529.175006:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654943360:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757529.175021:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757529.175029:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757529/real 1787757529] req@ffff8c05a9f7aa00 x1874588654943360/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757529.175037:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588654943360/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757529.175048:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757529.175051:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757529.175053:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654943360:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757529.302031:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757529.750083:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757530.326045:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757530.774111:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757531.349968:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757531.798060:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757532.373967:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757532.822224:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757533.398108:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757533.846199:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757534.294050:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757534.294063:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757534.294069:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757534.294082:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10590 00000100:00080000:3.0:1787757534.294096:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757534.294104:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757534.294156:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588654943488/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757534.294220:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757534.294225:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757534.294227:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757534.294237:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10590 00000100:00080000:3.0:1787757534.294242:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757534.294247:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757534.294259:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757534.294277:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588654943616/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757534.294290:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757534.294295:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757534.294297:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757534.294304:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10590 00000100:00080000:3.0:1787757534.294309:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757534.294312:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757534.294319:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757534.294332:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588654943744/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757534.294353:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654943488:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757534.294354:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757534.294384:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757534.294387:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757534.294393:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10590 00000100:00080000:3.0:1787757534.294397:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757534.294402:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757534.294411:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757534.294431:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588654943872/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757534.294443:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757534.294449:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757534.294451:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757534.294456:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10590 00000100:00080000:3.0:1787757534.294462:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757534.294469:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757534.294475:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757534.294492:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588654944000/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757534.294514:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757534.294518:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757534.294521:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757534.294526:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10590 00000100:00080000:3.0:1787757534.294531:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757534.294535:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757534.294557:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757534.294572:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588654944128/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757534.294585:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787757534.294589:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654943616:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757534.294592:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757534.294605:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757534.294614:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10590 00000100:00080000:3.0:1787757534.294620:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757534.294625:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757534.294632:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757534.294656:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588654944256/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757534.294668:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757534.294671:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757534.294674:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757534.294683:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10590 00000100:00100000:0.0:1787757534.294689:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654943744:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757534.294698:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757534.294707:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757534.294714:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757534.294738:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588654944384/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757534.294742:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654943872:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757534.294805:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654944000:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757534.294857:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654944128:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757534.295004:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654944256:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757534.295058:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654944384:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757534.422065:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757534.870333:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757535.446030:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757535.894213:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757536.469999:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757536.918230:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787757537.366124:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.107:988 00000400:00000100:1.0:1787757537.366163:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.107@tcp at host 192.168.201.107 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787757537.366176:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787757537.366208:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787757537.366214:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787757537.366221:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787757537.366228:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787757537.366234:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787757537.366241:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787757537.366249:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:0.0:1787757537.366265:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757534/real 1787757537] req@ffff8c05a9f7b480 x1874588654943744/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757537.366286:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588654943744/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757537.366295:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757537.366299:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757537.366302:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654943744:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757537.366307:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757534/real 1787757537] req@ffff8c05a9f78380 x1874588654943872/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757537.366315:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588654943872/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757537.366320:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757537.366323:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757537.366325:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654943872:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757537.366329:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757534/real 1787757537] req@ffff8c05a9f78a80 x1874588654944000/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757537.366336:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588654944000/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757537.366341:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757537.366343:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757537.366345:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654944000:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757537.366348:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757534/real 1787757537] req@ffff8c05a9f7a680 x1874588654944128/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757537.366355:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588654944128/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757537.366360:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757537.366363:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757537.366365:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654944128:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757537.366369:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757534/real 1787757537] req@ffff8c05a9f7a300 x1874588654944256/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757537.366382:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588654944256/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757537.366386:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757537.366388:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757537.366391:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654944256:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757537.366394:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757534/real 1787757537] req@ffff8c05a9f7aa00 x1874588654943488/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757537.366402:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588654943488/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757537.366406:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757537.366408:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757537.366410:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654943488:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787757537.366415:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757534/real 1787757537] req@ffff8c05a9f78700 x1874588654943616/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757537.366427:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588654943616/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757537.366435:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757537.366444:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757537.366448:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654943616:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757537.366486:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757534/real 1787757537] req@ffff8c05a9f79880 x1874588654944384/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757537.366499:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588654944384/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757537.366507:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757537.366510:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757537.366513:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654944384:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757537.493966:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757537.942098:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757538.518019:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757538.966301:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757539.414006:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757539.414024:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757539.414030:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757539.414046:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10595 00000100:00080000:3.0:1787757539.414065:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757539.414078:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757539.414144:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588654944512/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757539.414201:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757539.414209:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757539.414213:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757539.414222:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10595 00000100:00080000:3.0:1787757539.414231:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757539.414239:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757539.414260:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757539.414293:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588654944640/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757539.414315:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757539.414325:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757539.414329:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757539.414338:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10595 00000100:00080000:3.0:1787757539.414347:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757539.414354:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654944512:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757539.414365:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757539.414384:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757539.414422:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588654944768/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757539.414449:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757539.414476:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1787757539.414477:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757539.414489:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757539.414506:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10595 00000100:00080000:3.0:1787757539.414518:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757539.414529:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757539.414541:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757539.414573:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757539/real 1787757539] req@ffff8c05a9f7ad80 x1874588654944512/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757539.414605:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588654944896/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757539.414624:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588654944512/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757539.414640:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757539.414667:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757539.414676:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757539.414681:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757539.414693:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10595 00000100:00080000:0.0:1787757539.414704:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787757539.414710:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757539.414725:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654944512:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757539.414731:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757539.414755:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757539.414805:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654944640:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757539.414835:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588654945024/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757539.414860:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757539.414867:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757539.414875:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757539.414950:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10595 00000100:00080000:3.0:1787757539.414963:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757539.414974:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:0.0:1787757539.414980:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:3.0:1787757539.414988:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757539.415026:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588654945152/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757539.415033:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757539/real 1787757539] req@ffff8c05a9f7b800 x1874588654944640/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757539.415082:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757539.415092:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787757539.415100:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588654944640/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757539.415101:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757539.415126:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10595 00000100:00080000:3.0:1787757539.415145:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757539.415152:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757539.415163:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787757539.415175:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000008:00100000:3.0:1787757539.415187:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757539.415202:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654944640:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757539.415265:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654944768:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757539.415328:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757539.415359:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757539/real 1787757539] req@ffff8c05a9f78e00 x1874588654944768/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757539.415387:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588654945280/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757539.415393:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588654944768/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757539.415452:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757539.415470:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757539.415478:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654944768:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757539.415487:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787757539.415499:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654944896:192.168.201.107@tcp:8:kworker.0 00000100:00080000:3.0:1787757539.415501:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757539.415514:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757539.415528:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10595 00000100:00080000:3.0:1787757539.415541:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757539.415551:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757539.415564:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787757539.415579:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757539.415613:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588654945408/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757539.415630:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757539/real 1787757539] req@ffff8c05a9f79500 x1874588654944896/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757539.415672:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588654944896/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757539.415693:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757539.415701:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757539.415709:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654944896:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757539.415724:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654945024:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757539.415769:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757539.415823:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757539/real 1787757539] req@ffff8c05a9f7b100 x1874588654945024/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757539.415848:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588654945024/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757539.415864:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757539.415871:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757539.415926:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654945024:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757539.415940:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654945152:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757539.415982:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757539.416006:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757539/real 1787757539] req@ffff8c05a9f79180 x1874588654945152/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757539.416030:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588654945152/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757539.416058:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757539.416065:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757539.416072:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654945152:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757539.416112:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654945280:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757539.416167:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757539.416190:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757539/real 1787757539] req@ffff8c05a9f79f80 x1874588654945280/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757539.416224:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588654945280/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757539.416241:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757539.416248:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757539.416254:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654945280:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757539.416268:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654945408:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757539.416307:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757539.416332:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757539/real 1787757539] req@ffff8c05a9f7bb80 x1874588654945408/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757539.416357:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588654945408/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757539.416373:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757539.416380:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757539.416385:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654945408:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757539.542047:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757539.990170:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757540.566026:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757541.014268:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757541.589997:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757542.038132:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757542.614003:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757543.062331:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757543.638002:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757544.086148:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757544.534038:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757544.534055:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787757544.534061:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757544.534078:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10600 00000100:00080000:1.0:1787757544.534095:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757544.534108:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757544.534181:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588654945536/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757544.534243:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757544.534251:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757544.534253:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757544.534260:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10600 00000100:00080000:1.0:1787757544.534266:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757544.534273:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757544.534287:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757544.534308:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588654945664/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757544.534325:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757544.534331:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757544.534333:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757544.534340:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10600 00000100:00080000:1.0:1787757544.534344:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757544.534349:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757544.534354:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757544.534370:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588654945792/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757544.534379:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654945536:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757544.534389:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757544.534395:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757544.534398:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757544.534405:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10600 00000100:00080000:1.0:1787757544.534409:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757544.534414:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757544.534423:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757544.534437:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654945920/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757544.534451:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757544.534456:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757544.534459:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757544.534464:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10600 00000100:00080000:1.0:1787757544.534483:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757544.534489:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757544.534496:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787757544.534505:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757544.534519:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654946048/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757544.534534:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757544.534540:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757544.534542:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757544.534547:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10600 00000100:00080000:1.0:1787757544.534551:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757544.534556:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757544.534563:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757544.534583:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757544/real 1787757544] req@ffff8c05b2d9aa00 x1874588654945536/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757544.534585:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588654946176/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757544.534654:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757544.534662:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787757544.534669:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588654945536/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757544.534671:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757544.534711:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10600 00000100:00080000:1.0:1787757544.534723:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757544.534737:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787757544.534739:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787757544.534754:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787757544.534756:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757544.534774:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654945536:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787757544.534791:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654946304/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757544.534797:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654945664:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787757544.534824:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757544.534829:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757544.534834:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757544.534842:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10600 00000100:00080000:1.0:1787757544.534849:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787757544.534857:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757544.534858:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757544.534930:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757544.534932:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757544/real 1787757544] req@ffff8c05b2d9bb80 x1874588654945664/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757544.534962:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654946432/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757544.534967:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588654945664/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757544.535014:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757544.535022:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757544.535028:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654945664:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757544.535090:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654945792:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757544.535118:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757544.535143:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757544/real 1787757544] req@ffff8c05b2d9b100 x1874588654945792/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757544.535158:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654945792/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757544.535169:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757544.535173:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757544.535178:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654945792:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757544.535185:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654945920:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757544.535207:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757544.535219:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757544/real 1787757544] req@ffff8c05b2d99880 x1874588654945920/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757544.535234:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654945920/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757544.535244:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757544.535247:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757544.535250:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654945920:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757544.535257:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654946048:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757544.535276:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757544.535288:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757544/real 1787757544] req@ffff8c05b2d98380 x1874588654946048/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757544.535303:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654946048/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757544.535313:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757544.535316:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757544.535320:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654946048:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757544.535326:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654946176:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757544.535344:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757544.535355:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757544/real 1787757544] req@ffff8c05b2d9b480 x1874588654946176/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757544.535368:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654946176/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757544.535378:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757544.535382:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757544.535385:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654946176:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757544.535391:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654946304:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757544.535410:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757544.535421:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757544/real 1787757544] req@ffff8c05b2d98000 x1874588654946304/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757544.535435:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654946304/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757544.535454:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757544.535458:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757544.535462:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654946304:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757544.535468:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654946432:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757544.535485:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757544.535496:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757544/real 1787757544] req@ffff8c05b2d99f80 x1874588654946432/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757544.535508:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654946432/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757544.535518:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757544.535522:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757544.535526:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654946432:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757544.661999:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757545.110191:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757545.686015:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757546.134125:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757546.710018:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757547.158217:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757547.734039:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757548.182156:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757548.758045:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757549.206167:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757549.654088:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757549.654112:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757549.654121:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757549.654142:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10606 00000100:00080000:3.0:1787757549.654165:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757549.654177:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757549.654247:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588654946560/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757549.654306:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757549.654311:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757549.654314:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757549.654321:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10606 00000100:00080000:3.0:1787757549.654325:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757549.654330:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757549.654344:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757549.654361:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588654946688/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757549.654374:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757549.654379:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757549.654381:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757549.654386:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10606 00000100:00080000:3.0:1787757549.654390:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757549.654396:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757549.654401:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757549.654429:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588654946816/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757549.654445:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757549.654454:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787757549.654457:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654946560:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757549.654458:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757549.654474:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10606 00000100:00080000:3.0:1787757549.654480:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757549.654484:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757549.654491:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757549.654508:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588654946944/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757549.654522:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757549.654527:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757549.654529:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757549.654534:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10606 00000100:00080000:3.0:1787757549.654540:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757549.654545:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757549.654550:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757549.654572:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588654947072/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757549.654588:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757549.654617:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757549.654626:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757549.654629:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757549.654637:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10606 00000100:00080000:3.0:1787757549.654645:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757549.654651:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757549.654659:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757549.654678:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588654947200/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757549.654699:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757549.654703:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757549.654706:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757549.654712:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10606 00000100:00080000:3.0:1787757549.654717:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757549.654723:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757549/real 1787757549] req@ffff8c05a9f7bb80 x1874588654946560/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757549.654727:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757549.654741:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757549.654770:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588654947328/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757549.654785:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787757549.654786:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588654946560/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757549.654797:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757549.654804:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757549.654820:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10606 00000100:00080000:3.0:1787757549.654829:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757549.654832:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757549.654835:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757549.654847:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787757549.654850:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757549.654861:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654946560:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757549.654867:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588654947456/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757549.654944:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654946688:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757549.654991:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757549.655012:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757549/real 1787757549] req@ffff8c05a9f79f80 x1874588654946688/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757549.655029:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588654946688/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757549.655040:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757549.655044:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757549.655049:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654946688:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757549.655084:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654946816:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757549.655109:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757549.655123:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757549/real 1787757549] req@ffff8c05a9f79180 x1874588654946816/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757549.655139:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588654946816/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757549.655151:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757549.655155:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757549.655159:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654946816:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757549.655166:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654946944:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757549.655187:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757549.655201:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757549/real 1787757549] req@ffff8c05a9f7b100 x1874588654946944/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757549.655217:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588654946944/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757549.655228:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757549.655231:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757549.655234:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654946944:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757549.655241:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654947072:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757549.655262:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757549.655273:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757549/real 1787757549] req@ffff8c05a9f79500 x1874588654947072/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757549.655288:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588654947072/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757549.655298:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757549.655301:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757549.655305:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654947072:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757549.655312:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654947200:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757549.655331:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757549.655342:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757549/real 1787757549] req@ffff8c05a9f78e00 x1874588654947200/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757549.655356:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588654947200/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757549.655375:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757549.655379:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757549.655383:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654947200:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757549.655389:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654947328:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757549.655408:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757549.655419:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757549/real 1787757549] req@ffff8c05a9f7b800 x1874588654947328/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757549.655432:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588654947328/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757549.655443:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757549.655447:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757549.655450:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654947328:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757549.655456:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654947456:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757549.655475:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757549.655486:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757549/real 1787757549] req@ffff8c05a9f7ad80 x1874588654947456/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757549.655500:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588654947456/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757549.655510:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757549.655513:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757549.655518:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654947456:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757549.782034:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757550.230165:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757550.806025:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757551.254352:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757551.830012:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757552.278186:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757552.853995:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757553.302263:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757553.877988:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757554.326149:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757554.774004:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757554.774016:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757554.774022:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757554.774034:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10611 00000100:00080000:3.0:1787757554.774048:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757554.774058:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757554.774106:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588654947584/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757554.774145:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757554.774150:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757554.774152:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757554.774157:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10611 00000100:00080000:3.0:1787757554.774162:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757554.774166:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757554.774177:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757554.774191:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588654947712/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757554.774205:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757554.774211:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757554.774214:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757554.774219:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10611 00000100:00080000:3.0:1787757554.774223:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757554.774227:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757554.774233:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757554.774244:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588654947840/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757554.774261:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757554.774266:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757554.774268:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757554.774273:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10611 00000100:00080000:3.0:1787757554.774277:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757554.774290:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757554.774294:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654947584:192.168.201.107@tcp:250:kworker.0 00000008:00100000:3.0:1787757554.774298:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757554.774326:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588654947968/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757554.774341:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757554.774346:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757554.774348:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757554.774356:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10611 00000100:00080000:3.0:1787757554.774363:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757554.774368:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757554.774373:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757554.774395:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588654948096/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757554.774407:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757554.774411:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757554.774414:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757554.774419:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10611 00000100:00080000:3.0:1787757554.774428:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757554.774432:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757554.774439:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787757554.774450:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757554.774461:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588654948224/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757554.774476:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757554.774480:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757554.774482:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757554.774487:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10611 00000100:00080000:3.0:1787757554.774492:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757554.774497:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757554.774527:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757554.774548:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588654948352/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757554.774569:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757554.774574:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757554.774577:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757554.774586:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10611 00000100:00100000:0.0:1787757554.774599:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757554/real 1787757554] req@ffff8c05a9f79880 x1874588654947584/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757554.774599:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757554.774619:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757554.774646:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757554.774696:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588654947584/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757554.774697:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588654948480/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757554.774797:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757554.774809:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757554.774819:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654947584:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787757554.774835:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654947712:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757554.774951:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757554.774975:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757554/real 1787757554] req@ffff8c05a9f78700 x1874588654947712/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757554.774993:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588654947712/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757554.775005:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757554.775010:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757554.775014:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654947712:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757554.775056:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654947840:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757554.775085:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757554.775099:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757554/real 1787757554] req@ffff8c05a9f7aa00 x1874588654947840/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757554.775115:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588654947840/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757554.775125:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757554.775130:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757554.775134:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654947840:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757554.775142:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654947968:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757554.775176:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757554.775197:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757554/real 1787757554] req@ffff8c05a9f7a300 x1874588654947968/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757554.775223:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588654947968/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757554.775241:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757554.775248:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757554.775255:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654947968:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757554.775269:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654948096:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757554.775300:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757554.775316:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757554/real 1787757554] req@ffff8c05a9f7a680 x1874588654948096/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757554.775341:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588654948096/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757554.775352:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757554.775356:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757554.775360:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654948096:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757554.775368:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654948224:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757554.775387:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757554.775399:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757554/real 1787757554] req@ffff8c05a9f78a80 x1874588654948224/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757554.775414:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588654948224/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757554.775423:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757554.775427:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757554.775430:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654948224:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757554.775437:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654948352:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757554.775456:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757554.775467:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757554/real 1787757554] req@ffff8c05a9f78380 x1874588654948352/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757554.775481:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588654948352/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757554.775500:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757554.775535:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757554.775540:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654948352:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757554.775549:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654948480:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757554.775579:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757554.775593:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757554/real 1787757554] req@ffff8c05a9f7b480 x1874588654948480/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757554.775612:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588654948480/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757554.775629:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757554.775636:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757554.775643:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654948480:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757554.901989:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757555.350269:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757555.926037:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757556.374141:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757556.950033:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757557.398286:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757557.973996:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757558.422174:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757558.997971:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757559.446228:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757559.894009:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757559.894026:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757559.894033:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757559.894049:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10616 00000100:00080000:3.0:1787757559.894065:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757559.894076:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757559.894141:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588654948608/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757559.894187:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757559.894192:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757559.894195:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757559.894201:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10616 00000100:00080000:3.0:1787757559.894205:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757559.894210:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757559.894224:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757559.894241:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588654948736/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757559.894262:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757559.894267:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757559.894269:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757559.894273:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10616 00000100:00080000:3.0:1787757559.894277:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757559.894282:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757559.894287:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757559.894300:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588654948864/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757559.894314:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757559.894320:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757559.894322:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757559.894328:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10616 00000100:00080000:3.0:1787757559.894331:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757559.894336:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757559.894344:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757559.894361:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588654948992/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757559.894372:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654948608:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757559.894382:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757559.894392:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757559.894395:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757559.894400:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10616 00000100:00080000:3.0:1787757559.894406:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757559.894440:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757559.894449:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757559.894469:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588654949120/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757559.894482:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757559.894486:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757559.894489:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757559.894495:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10616 00000100:00080000:3.0:1787757559.894502:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757559.894507:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757559.894513:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787757559.894522:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757559.894538:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588654949248/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757559.894555:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757559.894559:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757559.894561:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757559.894568:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10616 00000100:00080000:3.0:1787757559.894578:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757559.894584:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757559.894591:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757559.894604:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588654949376/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757559.894617:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757559/real 1787757559] req@ffff8c05a9f7b480 x1874588654948608/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757559.894713:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588654948608/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757559.894717:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757559.894741:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757559.894747:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757559.894749:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757559.894760:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787757559.894772:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10616 00000100:00100000:0.0:1787757559.894773:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654948608:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757559.894799:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757559.894808:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654948736:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757559.894820:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757559.894838:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787757559.894852:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757559.894941:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588654949504/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757559.894956:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757559/real 1787757559] req@ffff8c05a9f78380 x1874588654948736/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757559.895006:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588654948736/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757559.895019:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757559.895025:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757559.895031:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654948736:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757559.895040:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654948864:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757559.895074:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757559.895088:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757559/real 1787757559] req@ffff8c05a9f78a80 x1874588654948864/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757559.895103:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588654948864/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757559.895113:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757559.895117:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757559.895127:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654948864:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757559.895165:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654948992:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757559.895203:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757559.895224:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757559/real 1787757559] req@ffff8c05a9f7a680 x1874588654948992/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757559.895239:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588654948992/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757559.895250:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757559.895255:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757559.895260:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654948992:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757559.895267:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654949120:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757559.895298:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757559.895312:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757559/real 1787757559] req@ffff8c05a9f7a300 x1874588654949120/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757559.895326:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588654949120/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757559.895336:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757559.895341:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757559.895351:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654949120:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757559.895368:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654949248:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757559.895442:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757559.895469:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757559/real 1787757559] req@ffff8c05a9f7aa00 x1874588654949248/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757559.895486:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588654949248/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757559.895502:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757559.895507:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757559.895512:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654949248:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757559.895531:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654949376:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757559.895570:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757559.895597:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757559/real 1787757559] req@ffff8c05a9f78700 x1874588654949376/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757559.895612:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588654949376/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757559.895622:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757559.895626:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757559.895630:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654949376:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757559.895638:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654949504:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757559.895657:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757559.895675:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757559/real 1787757559] req@ffff8c05a9f79880 x1874588654949504/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757559.895694:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588654949504/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757559.895706:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757559.895710:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757559.895717:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654949504:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757560.022014:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757560.470197:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757561.046010:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757561.494208:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757562.070025:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757562.518144:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757563.094004:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757563.542242:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757564.117999:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757564.566098:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757565.014065:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757565.014082:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787757565.014089:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757565.014106:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10621 00000100:00080000:1.0:1787757565.014125:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757565.014137:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757565.014206:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588654949632/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757565.014254:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757565.014259:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757565.014262:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757565.014268:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10621 00000100:00080000:1.0:1787757565.014273:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757565.014280:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757565.014296:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757565.014319:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588654949760/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757565.014352:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654949632:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757565.014364:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757565.014372:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757565.014375:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757565.014382:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10621 00000100:00080000:1.0:1787757565.014389:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757565.014395:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757565.014404:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757565.014427:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654949888/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757565.014441:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757565.014446:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757565.014448:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757565.014454:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10621 00000100:00080000:1.0:1787757565.014459:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757565.014464:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757565.014471:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787757565.014476:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757565.014494:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654950016/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757565.014510:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757565.014516:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757565.014518:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757565.014522:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10621 00000100:00080000:1.0:1787757565.014527:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757565.014532:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757565.014541:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757565.014555:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757565/real 1787757565] req@ffff8c05b2d9a300 x1874588654949632/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757565.014561:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588654950144/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757565.014614:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588654949632/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757565.014617:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757565.014640:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757565.014649:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757565.014652:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757565.014664:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787757565.014668:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10621 00000100:00100000:0.0:1787757565.014674:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654949632:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757565.014685:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757565.014700:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757565.014712:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757565.014720:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654949760:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757565.014753:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654950272/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757565.014763:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757565.014786:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757565/real 1787757565] req@ffff8c05b2d98700 x1874588654949760/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757565.014791:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757565.014814:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787757565.014822:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757565.014823:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654949760/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757565.014843:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10621 00000100:00080000:0.0:1787757565.014847:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757565.014855:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757565.014857:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757565.014868:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654949760:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787757565.014873:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757565.014917:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654949888:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757565.014954:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757565.014970:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757565/real 1787757565] req@ffff8c05b2d99f80 x1874588654949888/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787757565.014993:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757565.014999:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654949888/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757565.015016:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757565.015022:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757565.015030:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654949888:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757565.015041:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654950400/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757565.015054:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654950016:192.168.201.107@tcp:8:kworker.0 00000100:00080000:1.0:1787757565.015082:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757565.015088:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757565.015093:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787757565.015098:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757565.015106:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10621 00000100:00080000:1.0:1787757565.015117:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757565.015122:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757565/real 1787757565] req@ffff8c05b2d98000 x1874588654950016/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757565.015131:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757565.015151:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757565.015160:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654950016/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757565.015176:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787757565.015183:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588654950528/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757565.015189:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757565.015201:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654950016:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757565.015222:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654950144:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757565.015254:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757565.015291:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757565/real 1787757565] req@ffff8c05b2d9b480 x1874588654950144/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757565.015316:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654950144/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757565.015345:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757565.015349:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757565.015353:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654950144:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757565.015378:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654950272:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757565.015403:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757565.015417:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757565/real 1787757565] req@ffff8c05b2d98380 x1874588654950272/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757565.015432:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654950272/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757565.015442:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757565.015447:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757565.015451:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654950272:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757565.015457:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654950400:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757565.015476:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757565.015487:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757565/real 1787757565] req@ffff8c05b2d99880 x1874588654950400/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757565.015501:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654950400/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757565.015520:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757565.015524:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757565.015528:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654950400:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757565.015534:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654950528:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757565.015552:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757565.015563:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757565/real 1787757565] req@ffff8c05b2d9b100 x1874588654950528/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757565.015576:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654950528/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757565.015586:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757565.015590:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757565.015594:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654950528:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757565.142019:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757565.590258:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757566.165984:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757566.614130:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757567.189995:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757567.638218:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757568.213992:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757568.662126:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757569.238023:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757569.686320:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757570.133998:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757570.134012:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787757570.134018:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757570.134030:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10626 00000100:00080000:1.0:1787757570.134043:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757570.134053:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757570.134101:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588654950656/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757570.134139:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757570.134144:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757570.134147:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757570.134152:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10626 00000100:00080000:1.0:1787757570.134156:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757570.134161:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757570.134173:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757570.134193:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588654950784/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757570.134208:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757570.134214:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757570.134217:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757570.134223:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10626 00000100:00080000:1.0:1787757570.134230:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757570.134259:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757570.134270:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757570.134301:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654950656:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787757570.134303:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588654950912/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757570.134355:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757570.134362:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757570.134365:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757570.134372:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10626 00000100:00080000:1.0:1787757570.134378:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757570.134385:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757570.134395:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757570.134414:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654951040/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757570.134429:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757570.134434:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757570.134437:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757570.134442:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10626 00000800:00000100:0.0:1787757570.134453:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757570.134454:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757570.134473:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757570.134483:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757570.134500:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654951168/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757570.134515:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757570.134519:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757570.134522:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757570.134529:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10626 00000100:00080000:1.0:1787757570.134534:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757570.134540:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757570.134550:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757570.134564:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588654951296/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757570.134575:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757570/real 1787757570] req@ffff8c05b2d9bb80 x1874588654950656/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757570.134597:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757570.134613:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787757570.134618:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757570.134627:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588654950656/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757570.134632:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10626 00000100:00080000:1.0:1787757570.134653:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757570.134664:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757570.134672:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787757570.134682:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757570.134692:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654950656:192.168.201.107@tcp:250:kworker.0 00000008:00100000:1.0:1787757570.134706:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757570.134721:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654950784:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757570.134734:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654951424/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757570.134752:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757570.134756:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757570.134760:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757570.134766:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10626 00000100:00080000:1.0:1787757570.134784:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787757570.134798:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757570.134799:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757570.134818:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757570.134837:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654951552/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757570.134845:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757570/real 1787757570] req@ffff8c05b2d9aa00 x1874588654950784/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757570.135000:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588654950784/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757570.135013:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757570.135019:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757570.135024:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654950784:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757570.135068:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654950912:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757570.135104:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757570.135120:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757570/real 1787757570] req@ffff8c05b2d9b100 x1874588654950912/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757570.135136:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654950912/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757570.135147:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757570.135152:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757570.135156:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654950912:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757570.135166:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654951040:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757570.135205:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757570.135228:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757570/real 1787757570] req@ffff8c05b2d99880 x1874588654951040/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757570.135283:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654951040/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757570.135302:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757570.135308:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757570.135312:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654951040:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757570.135322:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654951168:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757570.135354:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757570.135370:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757570/real 1787757570] req@ffff8c05b2d98380 x1874588654951168/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757570.135385:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654951168/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757570.135396:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757570.135401:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757570.135404:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654951168:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757570.135412:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654951296:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757570.135432:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757570.135444:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757570/real 1787757570] req@ffff8c05b2d9b480 x1874588654951296/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757570.135478:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654951296/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757570.135495:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757570.135502:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757570.135509:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654951296:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757570.135522:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654951424:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757570.135564:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757570.135582:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757570/real 1787757570] req@ffff8c05b2d98000 x1874588654951424/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757570.135598:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654951424/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757570.135609:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757570.135614:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757570.135618:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654951424:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757570.135627:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654951552:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757570.135650:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757570.135661:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757570/real 1787757570] req@ffff8c05b2d99f80 x1874588654951552/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757570.135677:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654951552/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757570.135706:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757570.135713:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757570.135720:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654951552:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757570.262007:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757570.710172:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757571.286064:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757571.734192:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757572.310047:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757572.758077:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757573.334045:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757573.782307:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757574.358039:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757574.806187:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757575.254078:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757575.254091:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787757575.254097:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757575.254109:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10631 00000100:00080000:0.0:1787757575.254122:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757575.254131:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757575.254206:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588654951680/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757575.254248:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757575.254254:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757575.254258:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757575.254264:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10631 00000100:00080000:0.0:1787757575.254270:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757575.254276:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757575.254288:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757575.254305:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588654951808/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757575.254321:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757575.254326:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757575.254328:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757575.254333:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10631 00000100:00080000:0.0:1787757575.254338:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787757575.254344:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654951680:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757575.254346:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757575.254361:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757575.254381:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588654951936/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757575.254396:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757575.254401:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757575.254404:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757575.254409:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10631 00000100:00080000:0.0:1787757575.254414:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757575.254418:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757575.254424:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757575.254443:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588654952064/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787757575.254446:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757575.254471:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757575.254479:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757575.254483:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757575.254494:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10631 00000100:00080000:0.0:1787757575.254504:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787757575.254508:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757575/real 1787757575] req@ffff8c05a9f76300 x1874588654951680/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757575.254520:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757575.254539:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757575.254547:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588654951680/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757575.254566:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787757575.254572:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588654952192/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757575.254588:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757575.254596:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787757575.254598:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757575.254606:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757575.254610:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654951680:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757575.254618:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10631 00000100:00080000:0.0:1787757575.254628:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787757575.254631:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654951808:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757575.254641:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757575.254652:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757575.254674:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588654952320/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787757575.254684:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757575.254702:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757575.254707:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787757575.254712:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757575/real 1787757575] req@ffff8c05a9f75180 x1874588654951808/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757575.254713:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757575.254727:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10631 00000100:00080000:0.0:1787757575.254738:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757575.254748:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757575.254755:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588654951808/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787757575.254759:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787757575.254780:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787757575.254788:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588654952448/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757575.254789:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757575.254803:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654951808:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757575.254834:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757575.254839:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757575.254842:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757575.254847:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654951936:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757575.254850:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10631 00000100:00080000:0.0:1787757575.254864:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757575.254871:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:3.0:1787757575.254938:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:0.0:1787757575.254961:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757575.254962:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757575/real 1787757575] req@ffff8c05a9f77100 x1874588654951936/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757575.254991:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588654952576/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757575.255005:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588654951936/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757575.255021:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757575.255026:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757575.255032:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654951936:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757575.255043:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654952064:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757575.255073:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757575.255087:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757575/real 1787757575] req@ffff8c05a9f75500 x1874588654952064/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757575.255101:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588654952064/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757575.255111:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757575.255115:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757575.255118:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654952064:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757575.255125:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654952192:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757575.255168:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757575.255183:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757575/real 1787757575] req@ffff8c05a9f75880 x1874588654952192/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757575.255197:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588654952192/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757575.255215:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757575.255219:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757575.255223:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654952192:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757575.255230:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654952320:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757575.255253:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757575.255264:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757575/real 1787757575] req@ffff8c05a9f74000 x1874588654952320/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757575.255278:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588654952320/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757575.255288:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757575.255292:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757575.255295:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654952320:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757575.255302:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654952448:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757575.255321:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757575.255333:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757575/real 1787757575] req@ffff8c05a9f74e00 x1874588654952448/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757575.255347:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588654952448/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757575.255356:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757575.255359:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757575.255364:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654952448:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757575.255387:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654952576:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757575.255407:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757575.255419:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757575/real 1787757575] req@ffff8c05a9f76a00 x1874588654952576/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757575.255433:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588654952576/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757575.255443:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757575.255446:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757575.255450:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654952576:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757575.382019:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757575.830213:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757576.406055:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757576.854155:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757577.430065:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757577.878226:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757578.454017:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757578.902163:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757579.478050:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757579.926203:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757580.374009:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757580.374020:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787757580.374026:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757580.374037:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10636 00000100:00080000:0.0:1787757580.374073:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757580.374084:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757580.374131:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588654952704/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757580.374168:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757580.374176:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757580.374180:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757580.374188:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10636 00000100:00080000:0.0:1787757580.374196:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757580.374204:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757580.374219:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757580.374244:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588654952832/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757580.374246:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654952704:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757580.374265:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757580.374271:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757580.374274:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757580.374281:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10636 00000100:00080000:0.0:1787757580.374286:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757580.374292:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757580.374299:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757580.374318:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588654952960/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757580.374335:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787757580.374337:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757580.374342:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757580.374345:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757580.374351:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10636 00000100:00080000:0.0:1787757580.374356:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757580.374363:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757580.374372:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757580.374390:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588654953088/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757580.374397:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757580/real 1787757580] req@ffff8c05a9f76d80 x1874588654952704/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757580.374408:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757580.374414:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757580.374418:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757580.374424:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10636 00000100:00100000:3.0:1787757580.374429:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588654952704/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757580.374430:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757580.374438:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757580.374447:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787757580.374448:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757580.374456:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757580.374464:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588654953216/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757580.374464:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654952704:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757580.374480:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757580.374486:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757580.374489:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757580.374495:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10636 00000100:00100000:3.0:1787757580.374499:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654952832:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757580.374501:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757580.374508:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757580.374516:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757580.374533:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588654953344/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787757580.374544:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757580.374553:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757580.374559:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787757580.374563:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757580.374565:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757580/real 1787757580] req@ffff8c05a9f74a80 x1874588654952832/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757580.374569:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10636 00000100:00080000:0.0:1787757580.374576:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757580.374582:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757580.374589:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588654952832/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787757580.374591:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787757580.374607:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787757580.374609:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588654953472/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757580.374613:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757580.374621:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654952832:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757580.374627:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787757580.374632:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654952960:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757580.374634:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757580.374637:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757580.374643:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10636 00000100:00080000:0.0:1787757580.374649:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757580.374656:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757580.374664:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787757580.374665:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757580.374683:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588654953600/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757580.374685:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757580/real 1787757580] req@ffff8c05a9f74380 x1874588654952960/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757580.374709:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588654952960/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757580.374726:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757580.374733:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757580.374739:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654952960:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757580.374751:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654953088:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757580.374790:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757580.374808:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757580/real 1787757580] req@ffff8c05a9f77800 x1874588654953088/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757580.374832:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588654953088/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757580.374843:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757580.374847:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757580.374851:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654953088:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757580.374858:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654953216:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757580.374922:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757580.374938:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757580/real 1787757580] req@ffff8c05a9f75c00 x1874588654953216/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757580.374953:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588654953216/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757580.374964:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757580.374968:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757580.374972:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654953216:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757580.374994:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654953344:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757580.375017:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757580.375028:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757580/real 1787757580] req@ffff8c05a9f76680 x1874588654953344/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757580.375043:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588654953344/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757580.375087:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757580.375091:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757580.375095:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654953344:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757580.375102:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654953472:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757580.375125:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757580.375137:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757580/real 1787757580] req@ffff8c05a9f77b80 x1874588654953472/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757580.375151:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588654953472/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757580.375161:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757580.375165:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757580.375168:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654953472:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757580.375175:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654953600:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757580.375193:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757580.375203:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757580/real 1787757580] req@ffff8c05a9f76d80 x1874588654953600/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757580.375217:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588654953600/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757580.375226:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757580.375229:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757580.375234:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654953600:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757580.501990:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757580.950187:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757581.526070:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757581.974247:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757582.549996:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757582.998150:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757583.574019:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757584.022204:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757584.598049:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757585.046164:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757585.494082:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757585.494095:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757585.494100:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757585.494113:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10641 00000100:00080000:3.0:1787757585.494128:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757585.494137:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757585.494185:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588654953728/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757585.494280:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757585.494288:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757585.494292:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757585.494312:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10641 00000100:00080000:3.0:1787757585.494327:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757585.494346:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757585.494366:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757585.494399:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654953728:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757585.494434:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588654953856/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757585.494482:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757585.494490:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757585.494494:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757585.494502:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10641 00000100:00080000:3.0:1787757585.494514:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757585.494523:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757585.494532:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757585.494560:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588654953984/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787757585.494561:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757585.494591:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757585.494600:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757585.494603:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757585.494610:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10641 00000100:00080000:3.0:1787757585.494615:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757585.494620:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757585.494628:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757585.494648:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757585/real 1787757585] req@ffff8c05a9f7ad80 x1874588654953728/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757585.494651:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588654954112/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757585.494704:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757585.494709:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757585.494712:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787757585.494718:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588654953728/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757585.494725:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10641 00000100:00080000:3.0:1787757585.494736:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757585.494749:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:1.0:1787757585.494754:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787757585.494757:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787757585.494766:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757585.494775:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654953728:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757585.494782:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588654954240/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757585.494806:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757585.494811:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757585.494814:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787757585.494822:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654953856:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757585.494823:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10641 00000100:00080000:3.0:1787757585.494835:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757585.494843:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757585.494851:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757585.494916:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588654954368/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757585.494932:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757585.494937:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757585.494940:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787757585.494951:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757585.494954:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10641 00000100:00080000:3.0:1787757585.494994:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757585.495003:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757585.495014:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757585.495036:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757585/real 1787757585] req@ffff8c05a9f7b800 x1874588654953856/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757585.495042:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588654954496/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757585.495092:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757585.495097:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757585.495100:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787757585.495104:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588654953856/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757585.495114:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10641 00000100:00080000:3.0:1787757585.495130:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757585.495136:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757585.495139:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:1.0:1787757585.495152:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000008:00100000:3.0:1787757585.495154:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757585.495163:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654953856:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757585.495178:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654953984:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757585.495190:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588654954624/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787757585.495229:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757585.495256:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757585/real 1787757585] req@ffff8c05a9f78e00 x1874588654953984/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757585.495277:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588654953984/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757585.495288:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757585.495295:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757585.495303:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654953984:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757585.495313:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654954112:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757585.495349:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757585.495367:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757585/real 1787757585] req@ffff8c05a9f79500 x1874588654954112/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757585.495384:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588654954112/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757585.495396:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757585.495401:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757585.495405:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654954112:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757585.495414:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654954240:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757585.495437:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757585.495453:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757585/real 1787757585] req@ffff8c05a9f7b100 x1874588654954240/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757585.495468:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588654954240/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757585.495481:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757585.495484:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757585.495488:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654954240:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757585.495518:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654954368:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757585.495540:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757585.495552:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757585/real 1787757585] req@ffff8c05a9f79180 x1874588654954368/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757585.495566:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588654954368/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757585.495575:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757585.495578:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757585.495583:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654954368:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757585.495590:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654954496:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757585.495609:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757585.495620:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757585/real 1787757585] req@ffff8c05a9f79f80 x1874588654954496/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757585.495633:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588654954496/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757585.495643:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757585.495646:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757585.495650:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654954496:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757585.495656:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654954624:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757585.495673:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757585.495683:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757585/real 1787757585] req@ffff8c05a9f7bb80 x1874588654954624/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757585.495697:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588654954624/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757585.495707:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757585.495710:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757585.495714:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654954624:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757585.622020:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757586.070193:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757586.646081:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757587.094120:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757587.670062:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757588.118264:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757588.694048:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757589.142094:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757589.718039:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757590.166267:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757590.614028:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757590.614045:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757590.614053:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757590.614069:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10647 00000100:00080000:3.0:1787757590.614088:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757590.614099:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757590.614168:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588654954752/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757590.614210:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757590.614216:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757590.614218:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757590.614225:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10647 00000100:00080000:3.0:1787757590.614229:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757590.614234:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757590.614249:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757590.614268:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588654954880/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757590.614291:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757590.614296:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757590.614300:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757590.614309:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10647 00000100:00080000:3.0:1787757590.614316:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787757590.614331:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654954752:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757590.614332:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757590.614354:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757590.614399:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588654955008/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757590.614427:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757590.614435:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757590.614438:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757590.614444:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10647 00000100:00080000:3.0:1787757590.614451:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757590.614457:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757590.614465:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787757590.614479:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757590.614489:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588654955136/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757590.614514:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757590.614523:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757590.614527:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757590.614538:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10647 00000100:00100000:1.0:1787757590.614543:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757590/real 1787757590] req@ffff8c05a9f7bb80 x1874588654954752/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757590.614551:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757590.614579:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757590.614590:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588654954752/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787757590.614597:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787757590.614628:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757590.614635:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757590.614642:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588654955264/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757590.614674:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654954752:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757590.614679:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757590.614688:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757590.614691:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787757590.614695:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654954880:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757590.614709:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10647 00000100:00080000:3.0:1787757590.614719:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757590.614726:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757590.614734:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787757590.614751:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757590.614762:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588654955392/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757590.614779:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787757590.614782:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757590/real 1787757590] req@ffff8c05a9f79f80 x1874588654954880/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757590.614785:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757590.614793:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757590.614820:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10647 00000100:00080000:3.0:1787757590.614837:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787757590.614847:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588654954880/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757590.614849:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757590.614868:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757590.614962:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588654955520/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757590.614988:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757590.614993:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757590.615005:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787757590.615007:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757590.615020:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787757590.615022:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654954880:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757590.615039:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10647 00000100:00080000:3.0:1787757590.615048:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757590.615055:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757590.615064:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757590.615079:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654955008:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757590.615081:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588654955648/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787757590.615140:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757590.615176:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757590/real 1787757590] req@ffff8c05a9f79180 x1874588654955008/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757590.615205:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588654955008/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757590.615222:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757590.615230:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757590.615236:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654955008:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757590.615251:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654955136:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757590.615292:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757590.615309:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757590/real 1787757590] req@ffff8c05a9f7b100 x1874588654955136/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757590.615325:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588654955136/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757590.615336:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757590.615341:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757590.615345:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654955136:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757590.615353:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654955264:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757590.615375:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757590.615387:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757590/real 1787757590] req@ffff8c05a9f79500 x1874588654955264/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757590.615401:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588654955264/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757590.615420:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757590.615424:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757590.615428:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654955264:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757590.615435:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654955392:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757590.615454:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757590.615465:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757590/real 1787757590] req@ffff8c05a9f78e00 x1874588654955392/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757590.615479:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588654955392/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757590.615489:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757590.615492:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757590.615496:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654955392:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757590.615503:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654955520:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757590.615521:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757590.615532:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757590/real 1787757590] req@ffff8c05a9f7b800 x1874588654955520/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757590.615547:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588654955520/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757590.615556:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757590.615560:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757590.615564:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654955520:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757590.615589:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654955648:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757590.615608:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757590.615621:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757590/real 1787757590] req@ffff8c05a9f7ad80 x1874588654955648/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757590.615635:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588654955648/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757590.615645:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757590.615650:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757590.615654:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654955648:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757590.742023:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757591.190069:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757591.765948:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757592.214308:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757592.789986:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757593.238200:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757593.814015:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757594.264226:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757594.838007:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757595.286113:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757595.734032:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757595.734044:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787757595.734050:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757595.734062:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10652 00000100:00080000:0.0:1787757595.734075:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757595.734086:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757595.734140:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588654955776/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757595.734193:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757595.734202:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757595.734206:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757595.734216:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10652 00000100:00080000:0.0:1787757595.734225:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787757595.734262:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654955776:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757595.734269:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757595.734293:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757595.734322:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588654955904/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757595.734350:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757595.734355:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757595.734358:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757595.734362:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10652 00000100:00080000:0.0:1787757595.734368:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757595.734374:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757595.734380:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787757595.734398:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757595.734400:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588654956032/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757595.734419:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757595.734424:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757595.734426:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757595.734432:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10652 00000100:00080000:0.0:1787757595.734436:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757595.734441:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757595.734448:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757595.734467:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757595/real 1787757595] req@ffff8c05a9f76a00 x1874588654955776/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757595.734470:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588654956160/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757595.734500:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757595.734505:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757595.734507:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757595.734512:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10652 00000100:00080000:0.0:1787757595.734516:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757595.734523:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757595.734528:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757595.734542:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588654956288/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757595.734569:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757595.734575:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757595.734578:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757595.734586:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10652 00000100:00080000:0.0:1787757595.734592:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757595.734600:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757595.734610:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757595.734640:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588654956416/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757595.734657:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757595.734661:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787757595.734664:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757595.734669:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10652 00000100:00080000:0.0:1787757595.734675:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757595.734680:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757595.734686:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757595.734702:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588654956544/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757595.734722:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757595.734726:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757595.734729:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757595.734732:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10652 00000100:00080000:0.0:1787757595.734736:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757595.734740:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757595.734745:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757595.734756:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588654956672/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757595.734861:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588654955776/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757595.734924:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757595.734932:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757595.735008:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654955776:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757595.735068:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654955904:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757595.735117:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757595.735147:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757595/real 1787757595] req@ffff8c05a9f74e00 x1874588654955904/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757595.735176:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588654955904/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757595.735194:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757595.735202:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757595.735209:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654955904:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757595.735222:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654956032:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757595.735268:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757595.735293:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757595/real 1787757595] req@ffff8c05a9f74000 x1874588654956032/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757595.735314:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588654956032/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757595.735325:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757595.735329:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757595.735334:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654956032:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757595.735342:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654956160:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757595.735371:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757595.735385:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757595/real 1787757595] req@ffff8c05a9f75880 x1874588654956160/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757595.735399:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588654956160/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757595.735410:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757595.735414:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757595.735418:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654956160:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757595.735425:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654956288:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757595.735445:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757595.735456:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757595/real 1787757595] req@ffff8c05a9f75500 x1874588654956288/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757595.735470:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588654956288/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757595.735480:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757595.735484:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757595.735490:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654956288:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757595.735502:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654956416:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757595.735540:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757595.735562:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757595/real 1787757595] req@ffff8c05a9f77100 x1874588654956416/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757595.735588:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588654956416/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757595.735604:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757595.735609:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757595.735614:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654956416:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757595.735622:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654956544:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757595.735651:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757595.735666:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757595/real 1787757595] req@ffff8c05a9f75180 x1874588654956544/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757595.735680:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588654956544/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757595.735690:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757595.735694:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757595.735698:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654956544:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757595.735704:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654956672:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757595.735723:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757595.735734:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757595/real 1787757595] req@ffff8c05a9f76300 x1874588654956672/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757595.735748:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588654956672/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757595.735758:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757595.735761:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757595.735765:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654956672:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757595.861979:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757596.310097:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757596.885942:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757597.334078:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757597.910161:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757598.358332:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757598.933991:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757599.382255:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757599.958007:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757600.406284:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757600.854046:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757600.854059:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757600.854065:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757600.854078:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10657 00000100:00080000:3.0:1787757600.854091:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757600.854101:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757600.854147:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588654956800/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757600.854188:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757600.854193:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757600.854196:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757600.854201:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10657 00000100:00080000:3.0:1787757600.854205:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757600.854210:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757600.854221:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757600.854239:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588654956928/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757600.854258:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757600.854265:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757600.854269:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787757600.854277:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654956800:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757600.854282:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10657 00000100:00080000:3.0:1787757600.854298:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757600.854307:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757600.854319:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757600.854350:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588654957056/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757600.854371:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757600.854377:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757600.854379:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757600.854386:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10657 00000100:00080000:3.0:1787757600.854391:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757600.854398:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757600.854410:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757600.854430:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654956928:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757600.854441:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588654957184/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757600.854467:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757600.854476:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757600.854481:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787757600.854487:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654957056:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757600.854499:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10657 00000100:00080000:3.0:1787757600.854509:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757600.854516:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757600.854527:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757600.854550:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588654957312/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757600.854558:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654957184:192.168.201.107@tcp:8:kworker.0 00000100:00080000:3.0:1787757600.854575:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757600.854582:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757600.854585:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757600.854598:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10657 00000100:00080000:3.0:1787757600.854610:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757600.854616:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757600.854626:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757600.854648:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654957312:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757600.854663:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588654957440/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757600.854711:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757600.854719:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757600.854724:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787757600.854732:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654957440:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757600.854741:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10657 00000100:00080000:3.0:1787757600.854755:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757600.854762:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757600.854773:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757600.854797:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588654957568/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757600.854824:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757600.854830:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757600.854833:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757600.854838:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10657 00000100:00080000:3.0:1787757600.854842:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757600.854846:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757600.854853:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757600.854916:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588654957696/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757600.854930:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654957568:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757600.854986:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654957696:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757600.981996:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757601.430195:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757602.006018:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757602.454234:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757603.030003:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757603.478105:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787757603.926226:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.107:988 00000400:00000100:2.0:1787757603.926281:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.107@tcp at host 192.168.201.107 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787757603.926303:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787757603.926407:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787757603.926418:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787757603.926427:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787757603.926435:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787757603.926444:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787757603.926452:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787757603.926460:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:1.0:1787757603.926559:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757600/real 1787757603] req@ffff8c05a9f7b480 x1874588654957696/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757603.926609:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588654957696/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757603.926633:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757603.926670:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757603.926676:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654957696:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757603.926691:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757600/real 1787757603] req@ffff8c05a9f78380 x1874588654957568/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757603.926718:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588654957568/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757603.926734:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757603.926741:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757603.926748:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654957568:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757603.926762:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757600/real 1787757603] req@ffff8c05a9f78a80 x1874588654957440/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757603.926788:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588654957440/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757603.926804:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757603.926812:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757603.926819:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654957440:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757603.926832:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757600/real 1787757603] req@ffff8c05a9f7a680 x1874588654957312/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757603.926857:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588654957312/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757603.926919:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757603.926940:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757603.926948:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654957312:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757603.926961:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757600/real 1787757603] req@ffff8c05a9f7aa00 x1874588654957056/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757603.926987:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588654957056/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757603.927003:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757603.927010:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757603.927016:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654957056:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757603.927030:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757600/real 1787757603] req@ffff8c05a9f7a300 x1874588654957184/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757603.927054:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588654957184/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757603.927070:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757603.927078:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757603.927084:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654957184:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757603.927097:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757600/real 1787757603] req@ffff8c05a9f79880 x1874588654956800/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757603.927122:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588654956800/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757603.927138:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757603.927145:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757603.927152:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654956800:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787757603.927166:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757600/real 1787757603] req@ffff8c05a9f78700 x1874588654956928/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757603.927189:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588654956928/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757603.927205:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757603.927211:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757603.927217:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654956928:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757604.053996:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757604.502298:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757605.078025:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757605.526073:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757605.974050:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757605.974064:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787757605.974069:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757605.974083:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10662 00000100:00080000:0.0:1787757605.974096:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757605.974106:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757605.974152:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588654957824/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757605.974190:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757605.974195:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757605.974197:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757605.974203:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10662 00000100:00080000:0.0:1787757605.974208:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757605.974227:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757605.974239:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757605.974258:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588654957952/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757605.974277:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757605.974282:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757605.974286:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787757605.974286:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654957824:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757605.974302:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10662 00000100:00080000:0.0:1787757605.974316:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757605.974321:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757605.974328:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757605.974360:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588654958080/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757605.974382:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757605.974390:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757605.974394:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757605.974404:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10662 00000100:00080000:0.0:1787757605.974434:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757605.974444:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757605.974455:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757605.974479:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588654958208/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757605.974499:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757605.974506:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757605.974509:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757605.974516:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10662 00000100:00080000:0.0:1787757605.974523:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757605.974530:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757605.974539:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757605.974559:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588654958336/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757605.974577:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757605.974583:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757605.974587:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757605.974593:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10662 00000100:00080000:0.0:1787757605.974631:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787757605.974644:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757605.974648:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757605.974663:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757605.974691:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588654958464/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757605.974709:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757605.974714:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787757605.974718:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757605.974727:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10662 00000100:00080000:0.0:1787757605.974736:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787757605.974740:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757605/real 1787757605] req@ffff8c05a9f76300 x1874588654957824/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757605.974750:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757605.974766:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757605.974801:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588654957824/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757605.974802:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588654958592/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757605.974945:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757605.974955:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757605.974966:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757605.974968:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757605.974988:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787757605.974990:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10662 00000100:00080000:0.0:1787757605.975008:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787757605.975011:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654957824:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757605.975023:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757605.975038:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757605.975067:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588654958720/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757605.975072:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654957952:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757605.975166:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757605.975198:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757605/real 1787757605] req@ffff8c05a9f75180 x1874588654957952/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757605.975225:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588654957952/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757605.975243:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757605.975250:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757605.975258:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654957952:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757605.975271:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654958080:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757605.975315:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757605.975340:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757605/real 1787757605] req@ffff8c05a9f77100 x1874588654958080/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757605.975366:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588654958080/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757605.975381:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757605.975388:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757605.975394:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654958080:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757605.975407:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654958208:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757605.975447:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757605.975469:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757605/real 1787757605] req@ffff8c05a9f75500 x1874588654958208/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757605.975491:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588654958208/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757605.975508:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757605.975514:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757605.975521:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654958208:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757605.975533:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654958336:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757605.975573:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757605.975628:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757605/real 1787757605] req@ffff8c05a9f75880 x1874588654958336/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757605.975658:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588654958336/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757605.975675:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757605.975682:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757605.975689:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654958336:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757605.975703:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654958464:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757605.975750:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757605.975779:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757605/real 1787757605] req@ffff8c05a9f74000 x1874588654958464/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757605.975805:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588654958464/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757605.975822:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757605.975828:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757605.975835:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654958464:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757605.975848:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654958592:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757605.975933:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757605.975958:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757605/real 1787757605] req@ffff8c05a9f74e00 x1874588654958592/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757605.975983:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588654958592/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757605.975999:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757605.976005:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757605.976012:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654958592:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757605.976059:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654958720:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757605.976102:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757605.976123:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757605/real 1787757605] req@ffff8c05a9f76a00 x1874588654958720/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757605.976146:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588654958720/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757605.976174:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757605.976181:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757605.976188:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654958720:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757606.102018:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757606.550223:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757607.126039:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757607.574238:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757608.150059:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757608.598200:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757609.173992:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757609.622150:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757610.198008:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757610.646087:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757611.094044:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757611.094053:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787757611.094056:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757611.094064:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10667 00000100:00080000:1.0:1787757611.094073:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757611.094079:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757611.094110:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588654958848/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757611.094135:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757611.094138:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757611.094144:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757611.094147:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10667 00000100:00080000:1.0:1787757611.094149:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757611.094152:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757611.094159:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757611.094169:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588654958976/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757611.094177:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757611.094182:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757611.094184:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757611.094184:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654958848:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757611.094192:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10667 00000100:00080000:1.0:1787757611.094195:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757611.094197:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757611.094200:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757611.094210:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654959104/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757611.094216:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757611.094219:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757611.094220:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757611.094223:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10667 00000100:00080000:1.0:1787757611.094225:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757611.094227:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757611.094231:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757611.094237:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654959232/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757611.094244:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757611.094246:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757611.094247:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757611.094250:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10667 00000100:00080000:1.0:1787757611.094252:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757611.094254:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757611.094257:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757611.094263:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588654959360/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757611.094269:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757611.094271:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757611.094274:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787757611.094275:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757611.094279:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10667 00000100:00080000:1.0:1787757611.094281:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757611.094283:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757611.094285:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757611.094291:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654959488/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757611.094297:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757611.094299:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787757611.094301:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757611.094304:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10667 00000100:00080000:1.0:1787757611.094309:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757611.094311:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757611.094314:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757611.094321:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654959616/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757611.094327:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757611.094329:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757611.094331:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757611.094336:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10667 00000100:00100000:3.0:1787757611.094338:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757611/real 1787757611] req@ffff8c05b2d98700 x1874588654958848/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757611.094339:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757611.094345:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757611.094351:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757611.094362:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588654959744/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757611.094367:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654958848/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757611.094379:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757611.094382:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757611.094387:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654958848:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757611.094418:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654958976:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757611.094437:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757611.094447:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757611/real 1787757611] req@ffff8c05b2d9a300 x1874588654958976/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757611.094455:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588654958976/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757611.094461:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757611.094463:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757611.094465:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654958976:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757611.094470:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654959104:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757611.094507:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757611.094563:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757611/real 1787757611] req@ffff8c05b2d99f80 x1874588654959104/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757611.094572:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654959104/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757611.094577:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757611.094579:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757611.094581:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654959104:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757611.094586:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654959232:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757611.094600:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757611.094605:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757611/real 1787757611] req@ffff8c05b2d98000 x1874588654959232/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757611.094613:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654959232/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757611.094618:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757611.094619:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757611.094621:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654959232:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757611.094625:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654959360:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757611.094635:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757611.094641:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757611/real 1787757611] req@ffff8c05b2d9b480 x1874588654959360/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757611.094649:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654959360/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757611.094654:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757611.094655:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757611.094657:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654959360:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757611.094660:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654959488:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757611.094670:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757611.094676:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757611/real 1787757611] req@ffff8c05b2d98380 x1874588654959488/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757611.094683:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654959488/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757611.094688:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757611.094689:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757611.094691:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654959488:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757611.094694:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654959616:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757611.094704:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757611.094709:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757611/real 1787757611] req@ffff8c05b2d99880 x1874588654959616/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757611.094716:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654959616/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757611.094726:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757611.094728:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757611.094730:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654959616:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757611.094733:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654959744:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757611.094742:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757611.094748:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757611/real 1787757611] req@ffff8c05b2d9b100 x1874588654959744/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757611.094754:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654959744/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757611.094760:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757611.094762:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757611.094764:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654959744:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757611.222035:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757611.670130:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757612.246022:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757612.694275:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757613.269973:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757613.718060:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757614.293985:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757614.742183:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757615.318042:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757615.766158:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757616.213993:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757616.214006:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787757616.214012:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757616.214024:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10672 00000100:00080000:1.0:1787757616.214037:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757616.214047:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757616.214093:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588654959872/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757616.214130:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757616.214134:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757616.214137:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757616.214142:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10672 00000100:00080000:1.0:1787757616.214146:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757616.214151:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757616.214163:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757616.214183:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588654960000/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757616.214200:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787757616.214204:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654959872:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757616.214205:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757616.214211:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757616.214223:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10672 00000100:00080000:1.0:1787757616.214229:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757616.214233:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757616.214239:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757616.214257:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588654960128/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757616.214271:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757616.214276:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757616.214278:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757616.214283:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10672 00000100:00080000:1.0:1787757616.214290:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757616.214293:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757616.214300:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757616.214316:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654960256/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757616.214327:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757616.214331:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757616.214333:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757616.214339:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10672 00000100:00080000:1.0:1787757616.214345:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787757616.214348:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757616.214355:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757616.214364:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757616.214378:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654960384/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757616.214394:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757616.214401:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757616.214405:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757616.214414:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10672 00000100:00100000:3.0:1787757616.214449:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757616/real 1787757616] req@ffff8c05b2d9aa00 x1874588654959872/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757616.214453:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757616.214470:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757616.214489:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757616.214527:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588654960512/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757616.214531:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588654959872/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757616.214561:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757616.214572:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787757616.214575:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757616.214578:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757616.214585:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10672 00000100:00080000:1.0:1787757616.214595:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757616.214599:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787757616.214605:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757616.214616:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757616.214620:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654959872:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787757616.214637:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654960640/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757616.214666:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654960000:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787757616.214674:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757616.214682:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757616.214684:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757616.214691:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10672 00000100:00080000:1.0:1787757616.214695:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757616.214699:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757616.214706:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757616.214724:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654960768/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787757616.214741:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757616.214779:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757616/real 1787757616] req@ffff8c05b2d9bb80 x1874588654960000/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757616.214799:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588654960000/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757616.214811:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757616.214817:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757616.214824:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654960000:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757616.214836:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654960128:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757616.214939:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757616.214961:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757616/real 1787757616] req@ffff8c05b2d9b100 x1874588654960128/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757616.214979:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654960128/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757616.215001:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757616.215008:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757616.215016:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654960128:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757616.215028:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654960256:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757616.215073:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757616.215095:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757616/real 1787757616] req@ffff8c05b2d99880 x1874588654960256/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757616.215117:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654960256/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757616.215128:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757616.215132:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757616.215136:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654960256:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757616.215144:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654960384:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757616.215171:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757616.215184:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757616/real 1787757616] req@ffff8c05b2d98380 x1874588654960384/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757616.215206:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654960384/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757616.215222:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757616.215228:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757616.215236:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654960384:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757616.215248:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654960512:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757616.215288:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757616.215307:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757616/real 1787757616] req@ffff8c05b2d9b480 x1874588654960512/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757616.215323:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654960512/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757616.215333:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757616.215337:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757616.215341:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654960512:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757616.215368:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654960640:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757616.215393:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757616.215405:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757616/real 1787757616] req@ffff8c05b2d98000 x1874588654960640/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757616.215447:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654960640/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757616.215459:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757616.215463:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757616.215467:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654960640:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757616.215475:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654960768:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757616.215502:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757616.215517:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757616/real 1787757616] req@ffff8c05b2d99f80 x1874588654960768/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757616.215540:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654960768/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757616.215556:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757616.215562:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757616.215569:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654960768:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757616.342010:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757616.790237:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757617.366015:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757617.814179:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757618.390063:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757618.838320:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757619.414035:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757619.862232:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757620.438016:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757620.886304:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757621.334034:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757621.334052:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787757621.334058:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757621.334071:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10677 00000100:00080000:2.0:1787757621.334084:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757621.334094:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757621.334153:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588654960896/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757621.334210:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757621.334216:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757621.334218:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757621.334225:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10677 00000100:00080000:2.0:1787757621.334232:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757621.334238:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757621.334250:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757621.334273:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588654961024/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757621.334290:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757621.334296:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757621.334302:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757621.334304:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654960896:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757621.334312:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10677 00000100:00080000:2.0:1787757621.334322:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757621.334327:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757621.334386:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757621.334419:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588654961152/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787757621.334439:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757621.334449:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757621.334457:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757621.334459:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757621.334466:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10677 00000100:00080000:2.0:1787757621.334473:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757621.334478:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757621.334486:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757621.334503:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588654961280/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757621.334525:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757621.334529:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757621.334532:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757621.334538:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10677 00000100:00080000:2.0:1787757621.334543:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757621.334549:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757621.334555:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757621/real 1787757621] req@ffff8c05b2e73100 x1874588654960896/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787757621.334558:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757621.334585:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588654961408/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757621.334596:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588654960896/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757621.334612:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757621.334619:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757621.334625:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757621.334628:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757621.334636:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787757621.334642:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10677 00000100:00100000:3.0:1787757621.334650:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654960896:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757621.334656:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757621.334670:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757621.334682:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757621.334690:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654961024:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757621.334714:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588654961536/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787757621.334731:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757621.334759:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757621/real 1787757621] req@ffff8c05b2e72d80 x1874588654961024/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757621.334761:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757621.334778:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787757621.334784:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757621.334794:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10677 00000100:00100000:3.0:1787757621.334799:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588654961024/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757621.334804:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757621.334818:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:3.0:1787757621.334823:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787757621.334828:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787757621.334834:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757621.334843:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654961024:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757621.334858:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588654961664/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757621.334874:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757621.334924:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757621.334927:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757621.334933:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654961152:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757621.334935:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10677 00000100:00080000:2.0:1787757621.334948:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757621.334958:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757621.334969:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787757621.334984:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757621.335000:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588654961792/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757621.335013:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757621/real 1787757621] req@ffff8c05b2e72680 x1874588654961152/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757621.335059:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588654961152/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757621.335071:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757621.335076:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757621.335081:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654961152:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757621.335089:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654961280:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757621.335114:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757621.335127:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757621/real 1787757621] req@ffff8c05b2e70e00 x1874588654961280/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757621.335142:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588654961280/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757621.335152:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757621.335156:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757621.335160:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654961280:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757621.335167:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654961408:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757621.335187:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757621.335197:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757621/real 1787757621] req@ffff8c05b2e70000 x1874588654961408/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757621.335211:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588654961408/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757621.335221:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757621.335224:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757621.335228:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654961408:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757621.335249:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654961536:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757621.335269:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757621.335281:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757621/real 1787757621] req@ffff8c05b2e72300 x1874588654961536/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757621.335295:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588654961536/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757621.335304:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757621.335308:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757621.335311:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654961536:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757621.335318:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654961664:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757621.335368:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757621.335384:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757621/real 1787757621] req@ffff8c05b2e72a00 x1874588654961664/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757621.335399:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588654961664/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757621.335409:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757621.335413:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757621.335417:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654961664:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757621.335423:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654961792:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757621.335464:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757621.335477:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757621/real 1787757621] req@ffff8c05b2e71180 x1874588654961792/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757621.335491:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588654961792/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757621.335501:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757621.335505:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757621.335509:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654961792:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757621.462023:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757621.910116:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757622.486027:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757622.934252:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757623.510013:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757623.958189:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757624.533978:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757624.982295:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757625.558040:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757626.006121:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757626.454093:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757626.454110:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757626.454115:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757626.454129:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10682 00000100:00080000:3.0:1787757626.454142:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757626.454152:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757626.454221:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588654961920/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757626.454289:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757626.454294:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757626.454297:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757626.454303:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10682 00000100:00080000:3.0:1787757626.454308:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757626.454314:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757626.454327:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757626.454349:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588654962048/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757626.454357:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654961920:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757626.454378:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757626.454385:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757626.454389:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757626.454399:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10682 00000100:00080000:3.0:1787757626.454407:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757626.454416:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757626.454426:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757626.454455:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588654962176/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787757626.454456:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757626.454506:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757626.454513:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757626.454517:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757626.454528:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10682 00000100:00080000:3.0:1787757626.454538:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757626.454547:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757626.454559:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757626.454584:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588654962304/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757626.454605:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787757626.454610:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757626/real 1787757626] req@ffff8c05a9f78700 x1874588654961920/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757626.454614:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757626.454620:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757626.454638:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10682 00000100:00080000:3.0:1787757626.454665:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787757626.454670:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588654961920/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757626.454679:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757626.454693:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787757626.454710:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787757626.454720:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588654962432/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757626.454730:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757626.454749:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654961920:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757626.454752:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757626.454770:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757626.454774:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757626.454784:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10682 00000100:00080000:3.0:1787757626.454794:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787757626.454802:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654962048:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757626.454806:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757626.454823:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757626.454852:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588654962560/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787757626.454858:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757626.454923:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757626.454929:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757626.454931:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757626.454939:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10682 00000100:00080000:3.0:1787757626.454945:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757626.454952:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757626.454959:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757626.454985:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757626/real 1787757626] req@ffff8c05a9f79880 x1874588654962048/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757626.454986:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588654962688/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757626.455047:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757626.455053:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757626.455061:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787757626.455063:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588654962048/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757626.455070:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10682 00000100:00080000:3.0:1787757626.455080:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757626.455091:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757626.455102:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787757626.455109:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787757626.455121:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588654962816/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757626.455128:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757626.455147:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654962048:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757626.455177:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654962176:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757626.455218:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757626.455235:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757626/real 1787757626] req@ffff8c05a9f7a300 x1874588654962176/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757626.455282:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588654962176/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757626.455294:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757626.455298:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757626.455303:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654962176:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757626.455311:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654962304:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757626.455338:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757626.455352:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757626/real 1787757626] req@ffff8c05a9f7aa00 x1874588654962304/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757626.455375:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588654962304/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757626.455385:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757626.455389:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757626.455392:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654962304:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757626.455399:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654962432:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757626.455419:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757626.455430:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757626/real 1787757626] req@ffff8c05a9f7a680 x1874588654962432/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757626.455445:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588654962432/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757626.455454:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757626.455458:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757626.455461:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654962432:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757626.455483:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654962560:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757626.455504:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757626.455515:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757626/real 1787757626] req@ffff8c05a9f78a80 x1874588654962560/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757626.455530:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588654962560/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757626.455557:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757626.455564:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757626.455571:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654962560:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757626.455583:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654962688:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757626.455622:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757626.455644:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757626/real 1787757626] req@ffff8c05a9f78380 x1874588654962688/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757626.455659:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588654962688/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757626.455673:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757626.455679:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757626.455686:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654962688:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757626.455698:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654962816:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757626.455738:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757626.455760:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757626/real 1787757626] req@ffff8c05a9f7b480 x1874588654962816/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757626.455783:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588654962816/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757626.455794:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757626.455797:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757626.455802:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654962816:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757626.582017:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757627.030279:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757627.606051:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757628.054139:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757628.630039:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757629.078345:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757629.654059:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757630.102105:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757630.678030:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757631.126246:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757631.574041:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757631.574058:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787757631.574064:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757631.574080:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10687 00000100:00080000:0.0:1787757631.574096:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757631.574108:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757631.574195:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588654962944/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757631.574257:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757631.574266:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757631.574269:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757631.574278:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10687 00000100:00080000:0.0:1787757631.574288:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757631.574299:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757631.574317:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654962944:192.168.201.107@tcp:250:kworker.0 00000008:00100000:0.0:1787757631.574318:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757631.574351:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588654963072/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757631.574372:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757631.574377:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757631.574381:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757631.574389:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10687 00000100:00080000:0.0:1787757631.574410:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787757631.574417:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757631.574418:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757631.574427:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757631.574448:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588654963200/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757631.574466:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757631.574472:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757631.574476:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787757631.574476:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757631/real 1787757631] req@ffff8c05a9f76d80 x1874588654962944/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757631.574483:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10687 00000100:00080000:0.0:1787757631.574490:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757631.574499:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757631.574508:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757631.574511:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588654962944/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757631.574527:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588654963328/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757631.574532:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757631.574541:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787757631.574547:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787757631.574550:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654962944:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757631.574555:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757631.574560:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757631.574567:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10687 00000100:00080000:0.0:1787757631.574574:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757631.574582:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757631.574592:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757631.574595:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654963072:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757631.574616:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588654963456/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757631.574637:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787757631.574642:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757631.574644:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757631.574648:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757631.574655:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10687 00000100:00080000:0.0:1787757631.574661:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757631.574668:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757631.574669:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757631/real 1787757631] req@ffff8c05a9f77b80 x1874588654963072/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787757631.574677:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757631.574698:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588654963584/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757631.574710:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588654963072/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757631.574716:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757631.574723:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787757631.574727:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757631.574728:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757631.574735:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10687 00000100:00080000:1.0:1787757631.574736:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787757631.574744:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787757631.574745:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654963072:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757631.574752:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757631.574763:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757631.574764:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654963200:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757631.574786:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588654963712/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757631.574805:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787757631.574806:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757631.574812:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757631.574815:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787757631.574837:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757631/real 1787757631] req@ffff8c05a9f76680 x1874588654963200/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757631.574843:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10687 00000100:00080000:0.0:1787757631.574852:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757631.574861:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757631.574870:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588654963200/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787757631.574872:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787757631.574930:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757631.574938:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757631.574946:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654963200:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757631.574964:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654963328:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757631.574973:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588654963840/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787757631.575014:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757631.575045:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757631/real 1787757631] req@ffff8c05a9f75c00 x1874588654963328/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757631.575071:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588654963328/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757631.575088:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757631.575095:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757631.575101:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654963328:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757631.575128:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654963456:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757631.575190:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757631.575208:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757631/real 1787757631] req@ffff8c05a9f77800 x1874588654963456/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757631.575226:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588654963456/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757631.575237:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757631.575240:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757631.575244:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654963456:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757631.575252:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654963584:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757631.575274:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757631.575286:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757631/real 1787757631] req@ffff8c05a9f74380 x1874588654963584/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757631.575300:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588654963584/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757631.575310:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757631.575313:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757631.575316:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654963584:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757631.575323:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654963712:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757631.575341:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757631.575352:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757631/real 1787757631] req@ffff8c05a9f74a80 x1874588654963712/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757631.575366:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588654963712/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757631.575385:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757631.575389:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757631.575393:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654963712:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757631.575399:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654963840:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757631.575418:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757631.575428:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757631/real 1787757631] req@ffff8c05a9f76d80 x1874588654963840/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757631.575441:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588654963840/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757631.575451:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757631.575454:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757631.575458:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654963840:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757631.702007:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757632.150192:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757632.726086:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757633.174197:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757633.750047:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757634.198156:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757634.774025:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757635.222309:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757635.797996:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757636.246192:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757636.694022:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757636.694045:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787757636.694055:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757636.694097:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10693 00000100:00080000:0.0:1787757636.694119:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757636.694131:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757636.694204:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588654963968/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757636.694262:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757636.694271:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757636.694275:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757636.694286:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10693 00000100:00080000:0.0:1787757636.694295:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757636.694308:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757636.694329:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757636.694352:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654963968:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787757636.694368:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588654964096/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757636.694401:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757636.694408:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757636.694411:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757636.694417:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10693 00000100:00080000:0.0:1787757636.694444:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757636.694453:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:1.0:1787757636.694463:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:0.0:1787757636.694474:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757636.694512:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588654964224/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757636.694536:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787757636.694548:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757636/real 1787757636] req@ffff8c05a9f76a00 x1874588654963968/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757636.694551:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757636.694559:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757636.694582:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10693 00000100:00080000:0.0:1787757636.694603:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757636.694613:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757636.694628:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757636.694629:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588654963968/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757636.694662:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757636.694669:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757636.694683:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654963968:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787757636.694686:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588654964352/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757636.694716:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787757636.694725:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654964096:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757636.694730:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757636.694738:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757636.694747:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10693 00000100:00080000:0.0:1787757636.694755:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757636.694762:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757636.694772:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787757636.694785:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757636.694808:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588654964480/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757636.694816:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757636/real 1787757636] req@ffff8c05a9f74e00 x1874588654964096/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757636.694852:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757636.694869:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757636.694875:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757636.694931:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10693 00000100:00080000:0.0:1787757636.694938:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757636.694943:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757636.694951:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757636.694970:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588654964608/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757636.694984:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588654964096/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757636.695009:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757636.695014:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757636.695025:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787757636.695029:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787757636.695038:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787757636.695041:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654964096:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757636.695085:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10693 00000100:00080000:0.0:1787757636.695099:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787757636.695101:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654964224:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757636.695120:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757636.695132:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787757636.695151:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757636.695158:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588654964736/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757636.695174:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757636/real 1787757636] req@ffff8c05a9f74000 x1874588654964224/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757636.695177:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757636.695211:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757636.695219:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787757636.695222:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588654964224/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757636.695247:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10693 00000100:00080000:1.0:1787757636.695254:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757636.695262:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787757636.695264:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787757636.695271:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654964224:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757636.695285:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757636.695288:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654964352:192.168.201.107@tcp:8:kworker.0 00000008:00100000:0.0:1787757636.695303:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787757636.695325:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757636.695347:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588654964864/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757636.695349:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757636/real 1787757636] req@ffff8c05a9f75880 x1874588654964352/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757636.695458:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588654964352/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757636.695477:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757636.695486:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757636.695495:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654964352:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757636.695535:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654964480:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757636.695579:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757636.695598:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757636/real 1787757636] req@ffff8c05a9f75500 x1874588654964480/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757636.695614:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588654964480/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757636.695624:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757636.695629:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757636.695633:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654964480:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757636.695640:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654964608:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757636.695663:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757636.695676:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757636/real 1787757636] req@ffff8c05a9f77100 x1874588654964608/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757636.695690:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588654964608/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757636.695701:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757636.695705:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757636.695709:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654964608:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757636.695716:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654964736:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757636.695734:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757636.695745:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757636/real 1787757636] req@ffff8c05a9f75180 x1874588654964736/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757636.695759:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588654964736/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757636.695769:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757636.695772:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757636.695776:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654964736:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757636.695782:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654964864:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757636.695802:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757636.695813:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757636/real 1787757636] req@ffff8c05a9f76300 x1874588654964864/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757636.695828:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588654964864/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757636.695843:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757636.695849:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757636.695856:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654964864:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757636.821996:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757637.270318:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757637.846093:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757638.294096:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757638.870051:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757639.318219:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757639.894064:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757640.342139:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757640.918099:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757641.366295:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757641.814106:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757641.814125:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787757641.814132:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757641.814149:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10698 00000100:00080000:2.0:1787757641.814166:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757641.814178:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757641.814239:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588654964992/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757641.814284:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757641.814291:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757641.814294:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757641.814299:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10698 00000100:00080000:2.0:1787757641.814304:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757641.814308:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757641.814322:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757641.814342:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588654965120/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757641.814357:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757641.814376:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757641.814379:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757641.814385:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10698 00000100:00080000:2.0:1787757641.814390:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787757641.814397:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654964992:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757641.814399:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757641.814412:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757641.814429:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588654965248/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757641.814443:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757641.814449:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757641.814451:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757641.814456:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10698 00000100:00080000:2.0:1787757641.814461:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757641.814466:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757641.814474:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757641.814488:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588654965376/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757641.814500:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757641.814505:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757641.814507:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757641.814511:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10698 00000100:00080000:2.0:1787757641.814517:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757641.814522:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757641.814527:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787757641.814542:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757641.814545:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588654965504/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757641.814560:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757641.814565:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757641.814568:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757641.814575:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10698 00000100:00080000:2.0:1787757641.814581:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757641.814585:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757641.814591:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757641.814605:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588654965632/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757641.814616:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757641.814620:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787757641.814623:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757641.814628:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10698 00000100:00080000:2.0:1787757641.814633:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757641.814638:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757641.814646:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757641.814655:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757641/real 1787757641] req@ffff8c05b2e70a80 x1874588654964992/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757641.814667:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588654965760/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757641.814718:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757641.814723:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757641.814726:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757641.814732:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10698 00000100:00100000:1.0:1787757641.814740:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588654964992/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757641.814742:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757641.814757:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757641.814772:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787757641.814777:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757641.814790:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757641.814792:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588654965888/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757641.814802:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654964992:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787757641.814828:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654965120:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757641.814871:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757641.814935:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757641/real 1787757641] req@ffff8c05b2e71180 x1874588654965120/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757641.814952:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588654965120/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757641.814964:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757641.814995:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757641.815000:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654965120:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757641.815039:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654965248:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757641.815071:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757641.815086:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757641/real 1787757641] req@ffff8c05b2e72a00 x1874588654965248/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757641.815102:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588654965248/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757641.815113:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757641.815117:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757641.815121:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654965248:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757641.815128:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654965376:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757641.815148:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757641.815160:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757641/real 1787757641] req@ffff8c05b2e72300 x1874588654965376/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757641.815174:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588654965376/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757641.815184:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757641.815188:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757641.815191:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654965376:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757641.815198:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654965504:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757641.815218:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757641.815229:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757641/real 1787757641] req@ffff8c05b2e70000 x1874588654965504/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757641.815253:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588654965504/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757641.815264:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757641.815267:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757641.815270:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654965504:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757641.815277:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654965632:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757641.815297:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757641.815308:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757641/real 1787757641] req@ffff8c05b2e70e00 x1874588654965632/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757641.815323:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588654965632/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757641.815333:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757641.815336:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757641.815339:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654965632:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757641.815346:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654965760:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757641.815365:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757641.815381:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757641/real 1787757641] req@ffff8c05b2e72680 x1874588654965760/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757641.815404:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588654965760/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757641.815433:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757641.815439:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757641.815446:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654965760:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757641.815459:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654965888:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757641.815497:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757641.815514:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757641/real 1787757641] req@ffff8c05b2e72d80 x1874588654965888/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757641.815529:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588654965888/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757641.815540:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757641.815544:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757641.815549:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654965888:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757641.942047:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757642.390220:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757642.966066:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757643.414332:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757643.990021:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757644.438155:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757645.014018:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757645.462155:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757646.038017:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757646.486119:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757646.934066:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757646.934079:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757646.934085:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757646.934097:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10703 00000100:00080000:3.0:1787757646.934111:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757646.934121:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757646.934177:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588654966016/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757646.934230:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757646.934238:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757646.934242:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757646.934251:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10703 00000100:00080000:3.0:1787757646.934258:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757646.934264:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757646.934277:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757646.934298:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588654966144/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757646.934314:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757646.934319:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757646.934321:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757646.934326:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10703 00000100:00080000:3.0:1787757646.934330:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757646.934335:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757646.934341:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757646.934356:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588654966272/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757646.934383:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654966016:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757646.934387:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757646.934413:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757646.934418:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757646.934425:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10703 00000100:00080000:3.0:1787757646.934430:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757646.934433:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757646.934441:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757646.934468:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588654966400/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757646.934481:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757646.934486:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757646.934489:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757646.934494:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10703 00000100:00080000:3.0:1787757646.934502:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757646.934507:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757646.934513:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757646.934527:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588654966528/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757646.934540:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757646.934545:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757646.934548:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787757646.934561:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757646.934563:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10703 00000100:00080000:3.0:1787757646.934573:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757646.934580:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757646.934590:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757646.934625:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588654966656/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757646.934645:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787757646.934657:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757646/real 1787757646] req@ffff8c05a9f7ad80 x1874588654966016/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757646.934658:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757646.934668:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757646.934695:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10703 00000100:00080000:3.0:1787757646.934711:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757646.934724:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757646.934736:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757646.934751:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588654966016/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757646.934757:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588654966784/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757646.934823:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757646.934829:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757646.934834:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757646.934836:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757646.934846:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10703 00000100:00080000:1.0:1787757646.934857:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787757646.934857:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787757646.934978:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654966016:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757646.934979:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757646.935005:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757646.935017:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654966144:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757646.935032:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588654966912/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787757646.935069:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757646.935095:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757646/real 1787757646] req@ffff8c05a9f7b800 x1874588654966144/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757646.935114:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588654966144/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757646.935126:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757646.935131:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757646.935135:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654966144:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757646.935175:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654966272:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757646.935198:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757646.935210:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757646/real 1787757646] req@ffff8c05a9f78e00 x1874588654966272/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757646.935225:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588654966272/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757646.935235:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757646.935239:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757646.935244:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654966272:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757646.935252:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654966400:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757646.935274:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757646.935286:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757646/real 1787757646] req@ffff8c05a9f79500 x1874588654966400/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757646.935300:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588654966400/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757646.935350:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757646.935354:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757646.935358:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654966400:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757646.935366:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654966528:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757646.935395:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757646.935410:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757646/real 1787757646] req@ffff8c05a9f7b100 x1874588654966528/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757646.935426:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588654966528/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757646.935437:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757646.935441:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757646.935444:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654966528:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757646.935452:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654966656:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757646.935477:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757646.935489:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757646/real 1787757646] req@ffff8c05a9f79180 x1874588654966656/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757646.935513:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588654966656/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757646.935523:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757646.935527:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757646.935530:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654966656:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757646.935538:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654966784:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757646.935558:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757646.935570:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757646/real 1787757646] req@ffff8c05a9f79f80 x1874588654966784/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757646.935585:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588654966784/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757646.935595:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757646.935599:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757646.935602:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654966784:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757646.935609:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654966912:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757646.935628:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757646.935639:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757646/real 1787757646] req@ffff8c05a9f7bb80 x1874588654966912/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757646.935654:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588654966912/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757646.935683:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757646.935689:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757646.935696:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654966912:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757647.062030:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757647.510279:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757648.086024:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757648.534110:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757649.110064:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757649.558213:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757650.134048:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757650.582126:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757651.158014:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757651.606043:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757652.053928:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757652.053936:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757652.053939:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757652.053947:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10708 00000100:00080000:3.0:1787757652.053956:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757652.053962:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757652.053992:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588654967040/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757652.054014:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757652.054017:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757652.054018:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757652.054022:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10708 00000100:00080000:3.0:1787757652.054024:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757652.054026:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757652.054033:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757652.054042:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588654967168/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757652.054050:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757652.054052:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757652.054054:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757652.054058:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10708 00000100:00080000:3.0:1787757652.054060:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757652.054075:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757652.054079:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757652.054081:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654967040:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757652.054088:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588654967296/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757652.054097:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757652.054099:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757652.054100:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757652.054102:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10708 00000100:00080000:3.0:1787757652.054104:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757652.054107:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757652.054110:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757652.054118:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588654967424/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757652.054124:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757652.054127:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757652.054128:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757652.054147:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10708 00000100:00080000:3.0:1787757652.054150:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757652.054153:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757652.054156:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787757652.054168:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757652.054168:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588654967552/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757652.054177:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757652.054179:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757652.054181:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757652.054184:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10708 00000100:00080000:3.0:1787757652.054187:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757652.054189:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757652.054193:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757652.054201:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588654967680/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757652.054208:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787757652.054210:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757652/real 1787757652] req@ffff8c05a9f7bb80 x1874588654967040/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757652.054211:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757652.054215:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757652.054223:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10708 00000100:00080000:3.0:1787757652.054230:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787757652.054236:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588654967040/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757652.054236:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757652.054247:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787757652.054250:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757652.054255:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757652.054259:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654967040:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757652.054265:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588654967808/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757652.054268:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654967168:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757652.054284:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757652.054288:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1787757652.054292:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757652.054294:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757652.054302:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10708 00000100:00100000:1.0:1787757652.054305:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757652/real 1787757652] req@ffff8c05a9f79f80 x1874588654967168/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757652.054309:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757652.054319:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757652.054321:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588654967168/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787757652.054328:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787757652.054330:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757652.054335:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757652.054339:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654967168:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757652.054348:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588654967936/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757652.054363:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654967296:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757652.054379:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757652.054386:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757652/real 1787757652] req@ffff8c05a9f79180 x1874588654967296/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757652.054393:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588654967296/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757652.054399:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757652.054401:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757652.054404:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654967296:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757652.054407:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654967424:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757652.054417:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757652.054422:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757652/real 1787757652] req@ffff8c05a9f7b100 x1874588654967424/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757652.054429:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588654967424/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757652.054434:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757652.054436:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757652.054438:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654967424:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757652.054442:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654967552:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757652.054451:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757652.054474:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757652/real 1787757652] req@ffff8c05a9f79500 x1874588654967552/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757652.054483:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588654967552/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757652.054512:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757652.054514:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757652.054517:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654967552:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757652.054521:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654967680:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757652.054537:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757652.054546:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757652/real 1787757652] req@ffff8c05a9f78e00 x1874588654967680/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757652.054555:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588654967680/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757652.054562:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757652.054564:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757652.054566:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654967680:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757652.054570:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654967808:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757652.054598:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757652.054605:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757652/real 1787757652] req@ffff8c05a9f7b800 x1874588654967808/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757652.054612:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588654967808/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757652.054618:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757652.054620:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757652.054622:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654967808:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757652.054625:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654967936:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757652.054635:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757652.054641:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757652/real 1787757652] req@ffff8c05a9f7ad80 x1874588654967936/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757652.054647:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588654967936/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757652.054653:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757652.054655:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757652.054656:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654967936:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757652.181955:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757652.630128:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757653.206036:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757653.654151:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757654.230044:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757654.678284:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757655.254024:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757655.702139:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757656.278082:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757656.726079:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757657.173992:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757657.174001:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787757657.174004:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757657.174023:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10713 00000100:00080000:1.0:1787757657.174050:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757657.174059:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757657.174103:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588654968064/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757657.174140:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757657.174144:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757657.174146:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757657.174150:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10713 00000100:00080000:1.0:1787757657.174154:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757657.174158:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757657.174169:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757657.174193:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588654968192/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757657.174221:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654968064:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757657.174222:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757657.174230:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757657.174231:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757657.174235:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10713 00000100:00080000:1.0:1787757657.174237:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757657.174239:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757657.174243:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757657.174252:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654968320/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757657.174259:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757657.174262:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757657.174263:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757657.174266:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10713 00000100:00080000:1.0:1787757657.174269:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757657.174272:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757657.174276:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757657.174283:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654968448/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787757657.174287:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757657.174296:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757657.174299:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757657.174300:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757657.174304:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10713 00000100:00080000:1.0:1787757657.174308:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757657.174310:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757657.174313:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757657.174321:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588654968576/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757657.174327:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787757657.174329:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757657/real 1787757657] req@ffff8c05b2d9a300 x1874588654968064/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757657.174330:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757657.174334:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757657.174341:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10713 00000100:00080000:1.0:1787757657.174346:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757657.174352:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757657.174356:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757657.174368:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588654968064/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757657.174369:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654968704/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757657.174404:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757657.174412:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757657.174413:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787757657.174420:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787757657.174424:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757657.174424:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654968064:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757657.174433:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10713 00000100:00080000:1.0:1787757657.174436:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757657.174439:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757657.174443:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757657.174453:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654968832/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757657.174457:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654968192:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787757657.174467:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757657.174470:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757657.174472:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757657.174476:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10713 00000100:00080000:1.0:1787757657.174480:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757657.174482:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757657.174486:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787757657.174491:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757657.174499:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588654968960/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757657.174506:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757657/real 1787757657] req@ffff8c05b2d98700 x1874588654968192/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757657.174523:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654968192/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757657.174529:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757657.174532:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757657.174535:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654968192:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757657.174539:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654968320:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757657.174551:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757657.174557:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757657/real 1787757657] req@ffff8c05b2d99f80 x1874588654968320/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757657.174565:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654968320/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757657.174570:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757657.174572:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757657.174573:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654968320:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757657.174577:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654968448:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757657.174617:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757657.174628:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757657/real 1787757657] req@ffff8c05b2d98000 x1874588654968448/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757657.174642:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654968448/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757657.174675:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757657.174679:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757657.174683:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654968448:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757657.174688:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654968576:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757657.174748:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757657.174756:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757657/real 1787757657] req@ffff8c05b2d9b480 x1874588654968576/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757657.174764:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654968576/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757657.174769:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757657.174771:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757657.174773:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654968576:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757657.174777:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654968704:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757657.174789:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757657.174795:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757657/real 1787757657] req@ffff8c05b2d98380 x1874588654968704/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757657.174802:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654968704/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757657.174806:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757657.174808:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757657.174811:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654968704:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757657.174824:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654968832:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757657.174835:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757657.174841:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757657/real 1787757657] req@ffff8c05b2d99880 x1874588654968832/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757657.174848:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654968832/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757657.174853:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757657.174855:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757657.174856:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654968832:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757657.174860:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654968960:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757657.174869:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757657.174875:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757657/real 1787757657] req@ffff8c05b2d9b100 x1874588654968960/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757657.174882:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654968960/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757657.174887:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757657.174889:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757657.174908:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654968960:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757657.301989:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757657.750079:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757658.325954:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757658.774112:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757659.350061:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757659.798198:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757660.374021:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757660.822120:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757661.398030:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757661.846196:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757662.294036:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757662.294058:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787757662.294068:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757662.294085:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10718 00000100:00080000:0.0:1787757662.294104:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757662.294116:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757662.294181:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588654969088/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757662.294222:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757662.294227:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757662.294230:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757662.294236:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10718 00000100:00080000:0.0:1787757662.294241:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757662.294245:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757662.294260:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757662.294278:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588654969216/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757662.294292:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757662.294297:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757662.294300:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757662.294307:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10718 00000100:00080000:0.0:1787757662.294312:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757662.294319:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757662.294326:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757662.294340:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588654969344/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757662.294353:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654969088:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757662.294360:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757662.294366:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757662.294369:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757662.294388:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10718 00000100:00080000:0.0:1787757662.294393:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757662.294402:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757662.294420:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757662.294452:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588654969472/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757662.294473:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757662.294481:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757662.294485:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757662.294493:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10718 00000100:00080000:0.0:1787757662.294504:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757662.294514:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757662.294525:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787757662.294536:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757662.294560:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588654969600/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757662.294578:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757662.294583:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757662.294586:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757662.294594:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10718 00000100:00080000:0.0:1787757662.294602:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757662.294637:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757662.294649:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757662.294653:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757662/real 1787757662] req@ffff8c05a9f76300 x1874588654969088/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757662.294682:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588654969728/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757662.294706:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588654969088/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757662.294711:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757662.294730:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787757662.294737:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757662.294749:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10718 00000100:00080000:3.0:1787757662.294751:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757662.294757:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757662.294765:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787757662.294773:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757662.294782:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654969088:192.168.201.107@tcp:250:kworker.0 00000008:00100000:0.0:1787757662.294787:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757662.294800:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654969216:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757662.294815:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588654969856/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757662.294832:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757662.294836:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757662.294839:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757662.294845:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10718 00000100:00080000:0.0:1787757662.294850:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787757662.294854:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757662.294862:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757662.294872:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757662.294938:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757662/real 1787757662] req@ffff8c05a9f75180 x1874588654969216/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757662.294966:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588654969216/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757662.294980:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787757662.294984:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588654969984/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757662.294990:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757662.295004:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654969216:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757662.295062:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654969344:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757662.295098:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757662.295120:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757662/real 1787757662] req@ffff8c05a9f77100 x1874588654969344/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757662.295137:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588654969344/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757662.295148:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757662.295152:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757662.295156:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654969344:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757662.295164:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654969472:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757662.295184:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757662.295195:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757662/real 1787757662] req@ffff8c05a9f75500 x1874588654969472/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757662.295210:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588654969472/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757662.295220:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757662.295223:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757662.295227:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654969472:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757662.295234:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654969600:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757662.295268:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757662.295290:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757662/real 1787757662] req@ffff8c05a9f75880 x1874588654969600/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757662.295315:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588654969600/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757662.295347:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757662.295353:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757662.295360:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654969600:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757662.295369:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654969728:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757662.295400:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757662.295415:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757662/real 1787757662] req@ffff8c05a9f74000 x1874588654969728/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757662.295429:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588654969728/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757662.295440:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757662.295444:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757662.295448:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654969728:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757662.295454:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654969856:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757662.295473:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757662.295484:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757662/real 1787757662] req@ffff8c05a9f74e00 x1874588654969856/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757662.295498:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588654969856/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757662.295508:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757662.295511:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757662.295516:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654969856:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757662.295522:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654969984:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757662.295541:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757662.295551:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757662/real 1787757662] req@ffff8c05a9f76a00 x1874588654969984/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757662.295564:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588654969984/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757662.295574:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757662.295577:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757662.295581:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654969984:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757662.422035:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757662.870122:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757663.446033:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757663.894263:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757664.470000:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757664.918111:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757665.494071:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757665.942227:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757666.518021:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757666.966176:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757667.414040:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757667.414054:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787757667.414060:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757667.414073:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10723 00000100:00080000:0.0:1787757667.414088:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757667.414100:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757667.414150:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588654970112/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757667.414188:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757667.414193:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757667.414197:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757667.414204:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10723 00000100:00080000:0.0:1787757667.414211:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757667.414217:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757667.414230:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757667.414247:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654970112:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787757667.414259:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588654970240/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757667.414286:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757667.414291:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757667.414295:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757667.414301:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10723 00000100:00080000:0.0:1787757667.414306:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757667.414312:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757667.414318:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757667.414338:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588654970368/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757667.414354:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757667.414360:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757667.414363:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757667.414369:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10723 00000100:00080000:0.0:1787757667.414375:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757667.414382:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757667.414391:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757667.414422:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654970240:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757667.414423:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588654970496/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757667.414453:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757667.414458:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757667.414461:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757667.414468:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10723 00000100:00080000:0.0:1787757667.414475:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757667.414481:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757667.414489:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757667.414510:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588654970624/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757667.414543:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654970368:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757667.414548:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757667.414559:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757667.414565:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757667.414573:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10723 00000100:00080000:0.0:1787757667.414579:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757667.414585:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757667.414593:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757667.414613:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588654970752/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757667.414623:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654970496:192.168.201.107@tcp:8:kworker.0 00000100:00080000:0.0:1787757667.414635:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757667.414643:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787757667.414646:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757667.414655:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10723 00000100:00080000:0.0:1787757667.414662:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787757667.414670:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654970624:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757667.414672:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757667.414687:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757667.414709:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588654970880/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757667.414729:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654970752:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757667.414740:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757667.414747:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757667.414751:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757667.414766:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10723 00000100:00080000:0.0:1787757667.414771:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757667.414777:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757667.414784:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757667.414795:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654970880:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757667.414806:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588654971008/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757667.414938:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654971008:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757667.542013:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757667.990288:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757668.566015:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757669.014190:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757669.590028:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757670.038265:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787757670.486254:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.107:988 00000400:00000100:3.0:1787757670.486308:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.107@tcp at host 192.168.201.107 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787757670.486328:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:3.0:1787757670.486376:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:3.0:1787757670.486387:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:3.0:1787757670.486396:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:3.0:1787757670.486405:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:3.0:1787757670.486414:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:3.0:1787757670.486422:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:3.0:1787757670.486431:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:2.0:1787757670.486598:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757667/real 1787757670] req@ffff8c05a9f76a00 x1874588654971008/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757670.486679:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588654971008/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757670.486704:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757670.486715:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757670.486724:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654971008:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757670.486741:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757667/real 1787757670] req@ffff8c05a9f77b80 x1874588654970880/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757670.486765:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588654970880/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757670.486781:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757670.486787:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757670.486794:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654970880:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757670.486808:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757667/real 1787757670] req@ffff8c05a9f76680 x1874588654970752/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757670.486831:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588654970752/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757670.486848:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757670.486856:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757670.486863:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654970752:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757670.486974:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757667/real 1787757670] req@ffff8c05a9f77800 x1874588654970496/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757670.487005:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588654970496/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757670.487022:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757670.487029:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757670.487036:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654970496:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757670.487050:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757667/real 1787757670] req@ffff8c05a9f75c00 x1874588654970624/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757670.487072:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588654970624/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757670.487089:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757670.487096:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757670.487103:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654970624:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757670.487118:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757667/real 1787757670] req@ffff8c05a9f74a80 x1874588654970240/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757670.487144:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588654970240/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757670.487161:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757670.487168:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757670.487176:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654970240:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757670.487191:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757667/real 1787757670] req@ffff8c05a9f74380 x1874588654970368/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757670.487233:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588654970368/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757670.487249:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757670.487258:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757670.487269:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654970368:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757670.487285:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757667/real 1787757670] req@ffff8c05a9f76d80 x1874588654970112/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757670.487310:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588654970112/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757670.487323:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757670.487329:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757670.487335:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654970112:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757670.613983:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757671.062240:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757671.638032:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757672.086246:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757672.534101:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757672.534113:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787757672.534118:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757672.534131:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10728 00000100:00080000:1.0:1787757672.534143:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757672.534152:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757672.534197:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588654971136/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757672.534237:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757672.534242:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757672.534244:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757672.534250:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10728 00000100:00080000:1.0:1787757672.534254:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757672.534258:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757672.534270:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757672.534287:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588654971264/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757672.534301:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757672.534305:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757672.534308:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757672.534313:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10728 00000100:00080000:1.0:1787757672.534318:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757672.534324:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757672.534332:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757672.534344:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588654971392/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757672.534358:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757672.534362:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757672.534365:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757672.534369:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10728 00000100:00080000:1.0:1787757672.534373:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757672.534377:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757672.534382:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757672.534393:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654971520/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757672.534406:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757672.534412:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757672.534415:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787757672.534456:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654971136:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757672.534457:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10728 00000100:00080000:1.0:1787757672.534468:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757672.534486:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757672.534507:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757672.534544:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654971648/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757672.534565:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757672.534570:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757672.534576:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757672.534585:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10728 00000100:00080000:1.0:1787757672.534593:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757672.534601:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757672.534615:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787757672.534640:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757672.534646:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588654971776/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757672.534669:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757672.534675:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787757672.534681:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757672.534689:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10728 00000100:00080000:1.0:1787757672.534698:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757672.534707:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757672.534719:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757672.534745:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654971904/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757672.534760:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757672/real 1787757672] req@ffff8c05b2d9bb80 x1874588654971136/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757672.534773:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757672.534797:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757672.534808:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757672.534824:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10728 00000100:00080000:1.0:1787757672.534834:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787757672.534844:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588654971136/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757672.534853:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757672.534872:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787757672.534974:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757672.534993:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757672.535005:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654971136:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787757672.535019:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654972032/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757672.535041:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654971264:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757672.535120:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757672.535160:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757672/real 1787757672] req@ffff8c05b2d9aa00 x1874588654971264/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757672.535189:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588654971264/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757672.535208:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757672.535216:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757672.535223:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654971264:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757672.535238:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654971392:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757672.535287:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757672.535314:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757672/real 1787757672] req@ffff8c05b2d9b100 x1874588654971392/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757672.535341:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654971392/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757672.535373:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757672.535383:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757672.535393:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654971392:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757672.535486:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654971520:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757672.535548:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757672.535585:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757672/real 1787757672] req@ffff8c05b2d99880 x1874588654971520/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757672.535612:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654971520/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757672.535630:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757672.535637:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757672.535644:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654971520:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757672.535658:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654971648:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757672.535704:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757672.535729:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757672/real 1787757672] req@ffff8c05b2d98380 x1874588654971648/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757672.535752:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654971648/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757672.535769:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757672.535775:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757672.535783:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654971648:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757672.535795:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654971776:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757672.535836:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757672.535857:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757672/real 1787757672] req@ffff8c05b2d9b480 x1874588654971776/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757672.535941:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654971776/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757672.535959:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757672.535967:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757672.535974:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654971776:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757672.535987:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654971904:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757672.536033:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757672.536058:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757672/real 1787757672] req@ffff8c05b2d98000 x1874588654971904/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757672.536082:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654971904/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757672.536099:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757672.536106:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757672.536113:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654971904:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757672.536126:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654972032:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757672.536169:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757672.536194:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757672/real 1787757672] req@ffff8c05b2d99f80 x1874588654972032/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757672.536216:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654972032/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757672.536231:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757672.536237:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757672.536244:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654972032:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757672.661987:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757673.110214:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757673.685989:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757674.134321:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757674.710019:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757675.158187:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757675.733970:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757676.182310:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757676.758012:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757677.206179:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757677.654054:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757677.654076:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787757677.654085:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757677.654107:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10734 00000100:00080000:2.0:1787757677.654129:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757677.654146:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757677.654231:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588654972160/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757677.654275:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757677.654281:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757677.654284:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757677.654290:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10734 00000100:00080000:2.0:1787757677.654295:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757677.654301:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757677.654325:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757677.654377:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588654972288/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757677.654396:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787757677.654401:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654972160:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757677.654404:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757677.654413:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757677.654426:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10734 00000100:00080000:2.0:1787757677.654434:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757677.654442:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757677.654454:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757677.654486:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588654972416/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757677.654508:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757677.654513:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757677.654517:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757677.654523:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10734 00000100:00080000:2.0:1787757677.654529:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757677.654539:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:3.0:1787757677.654541:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:2.0:1787757677.654555:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757677.654577:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588654972544/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757677.654591:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757677.654596:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757677.654599:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757677.654604:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10734 00000100:00080000:2.0:1787757677.654610:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757677.654615:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757677.654623:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757677.654632:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757677/real 1787757677] req@ffff8c05b2e73100 x1874588654972160/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757677.654647:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588654972672/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757677.654670:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787757677.654672:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588654972160/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757677.654691:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757677.654702:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757677.654705:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757677.654716:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10734 00000100:00080000:3.0:1787757677.654717:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757677.654729:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654972160:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757677.654733:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757677.654760:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757677.654770:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757677.654771:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654972288:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757677.654802:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588654972800/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787757677.654820:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757677.654823:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757677.654830:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787757677.654834:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757677.654842:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10734 00000100:00100000:3.0:1787757677.654848:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757677/real 1787757677] req@ffff8c05b2e70700 x1874588654972288/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757677.654853:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757677.654867:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757677.654939:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588654972288/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787757677.654947:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787757677.654964:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757677.654972:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757677.654980:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588654972928/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757677.654985:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654972288:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757677.655011:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654972416:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757677.655034:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757677.655039:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757677.655042:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757677.655049:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10734 00000100:00080000:2.0:1787757677.655057:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787757677.655058:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757677.655074:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757677.655084:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757677/real 1787757677] req@ffff8c05b2e73480 x1874588654972416/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787757677.655089:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757677.655112:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588654972416/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757677.655124:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787757677.655129:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588654973056/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757677.655134:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757677.655148:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654972416:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757677.655169:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654972544:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757677.655201:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757677.655215:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757677/real 1787757677] req@ffff8c05b2e71500 x1874588654972544/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757677.655231:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588654972544/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757677.655260:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757677.655266:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757677.655273:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654972544:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757677.655286:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654972672:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757677.655326:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757677.655374:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757677/real 1787757677] req@ffff8c05b2e73b80 x1874588654972672/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757677.655390:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588654972672/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757677.655401:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757677.655406:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757677.655410:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654972672:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757677.655435:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654972800:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757677.655463:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757677.655476:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757677/real 1787757677] req@ffff8c05b2e71c00 x1874588654972800/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757677.655491:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588654972800/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757677.655501:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757677.655504:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757677.655509:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654972800:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757677.655516:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654972928:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757677.655536:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757677.655547:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757677/real 1787757677] req@ffff8c05b2e71880 x1874588654972928/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757677.655561:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588654972928/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757677.655571:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757677.655574:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757677.655577:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654972928:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757677.655584:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654973056:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757677.655602:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757677.655613:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757677/real 1787757677] req@ffff8c05b2e70380 x1874588654973056/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757677.655628:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588654973056/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757677.655638:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757677.655641:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757677.655645:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654973056:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757677.782056:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757678.230198:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757678.805969:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757679.254077:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757679.829995:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757680.278156:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757680.854020:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757681.302178:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757681.878001:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757682.326366:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757682.774080:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757682.774093:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787757682.774098:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757682.774110:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10739 00000100:00080000:0.0:1787757682.774124:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757682.774134:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757682.774181:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588654973184/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757682.774220:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757682.774225:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757682.774227:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757682.774233:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10739 00000100:00080000:0.0:1787757682.774237:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757682.774243:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757682.774284:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757682.774304:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588654973312/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757682.774321:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757682.774326:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757682.774329:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757682.774334:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10739 00000100:00080000:0.0:1787757682.774339:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757682.774345:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757682.774352:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757682.774356:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654973184:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787757682.774369:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588654973440/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757682.774386:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757682.774390:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757682.774392:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757682.774397:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10739 00000100:00080000:0.0:1787757682.774402:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757682.774406:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757682.774413:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757682.774428:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588654973568/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757682.774441:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757682.774447:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757682.774453:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787757682.774455:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757682.774463:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10739 00000100:00080000:0.0:1787757682.774470:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757682.774477:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757682.774484:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757682.774500:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588654973696/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757682.774512:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787757682.774520:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757682/real 1787757682] req@ffff8c05a9f74e00 x1874588654973184/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757682.774521:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757682.774527:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757682.774541:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10739 00000100:00080000:0.0:1787757682.774552:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757682.774560:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757682.774567:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757682.774570:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588654973184/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757682.774585:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588654973824/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757682.774591:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757682.774609:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787757682.774614:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787757682.774621:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654973184:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757682.774624:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787757682.774632:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757682.774642:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654973312:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757682.774644:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10739 00000100:00080000:0.0:1787757682.774656:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757682.774663:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757682.774669:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757682.774691:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588654973952/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787757682.774706:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757682.774723:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787757682.774729:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757682/real 1787757682] req@ffff8c05a9f74000 x1874588654973312/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757682.774734:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757682.774743:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757682.774762:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10739 00000100:00100000:3.0:1787757682.774764:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588654973312/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757682.774776:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757682.774781:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757682.774793:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787757682.774794:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757682.774802:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654973312:192.168.201.107@tcp:38:kworker.0 00000008:00100000:0.0:1787757682.774809:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757682.774836:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588654974080/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757682.774848:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654973440:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757682.774932:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757682.774953:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757682/real 1787757682] req@ffff8c05a9f75880 x1874588654973440/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757682.774969:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588654973440/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757682.774980:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757682.774984:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757682.774988:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654973440:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757682.774996:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654973568:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757682.775017:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757682.775028:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757682/real 1787757682] req@ffff8c05a9f75500 x1874588654973568/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757682.775042:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588654973568/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757682.775052:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757682.775055:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757682.775059:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654973568:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757682.775065:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654973696:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757682.775084:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757682.775095:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757682/real 1787757682] req@ffff8c05a9f77100 x1874588654973696/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757682.775110:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588654973696/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757682.775119:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757682.775123:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757682.775127:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654973696:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757682.775137:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654973824:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757682.775171:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757682.775193:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757682/real 1787757682] req@ffff8c05a9f75180 x1874588654973824/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757682.775219:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588654973824/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757682.775236:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757682.775242:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757682.775278:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654973824:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757682.775292:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654973952:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757682.775336:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757682.775360:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757682/real 1787757682] req@ffff8c05a9f76300 x1874588654973952/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757682.775384:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588654973952/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757682.775403:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757682.775407:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757682.775411:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654973952:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757682.775437:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654974080:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757682.775465:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757682.775478:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757682/real 1787757682] req@ffff8c05a9f76d80 x1874588654974080/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757682.775492:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588654974080/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757682.775502:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757682.775506:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757682.775510:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654974080:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757682.902042:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757683.350090:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757683.926011:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757684.374271:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757684.950006:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757685.398119:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757685.973991:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757686.422241:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757686.998095:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757687.446089:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757687.894062:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757687.894074:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787757687.894080:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757687.894091:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10744 00000100:00080000:1.0:1787757687.894104:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757687.894113:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757687.894185:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588654974208/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757687.894222:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757687.894227:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757687.894230:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757687.894235:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10744 00000100:00080000:1.0:1787757687.894239:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757687.894244:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757687.894256:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757687.894273:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588654974336/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757687.894287:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757687.894292:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757687.894294:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757687.894299:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10744 00000100:00080000:1.0:1787757687.894303:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757687.894308:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757687.894314:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757687.894331:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654974208:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787757687.894332:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654974464/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757687.894362:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757687.894366:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757687.894368:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757687.894372:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10744 00000100:00080000:1.0:1787757687.894376:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757687.894381:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757687.894387:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757687.894403:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654974592/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757687.894416:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757687.894421:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757687.894423:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757687.894428:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10744 00000100:00080000:1.0:1787757687.894434:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757687.894438:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757687.894446:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787757687.894456:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757687.894467:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588654974720/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757687.894481:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757687.894486:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757687.894489:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757687.894494:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10744 00000100:00080000:1.0:1787757687.894498:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757687.894502:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757687.894509:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757687.894525:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654974848/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757687.894538:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787757687.894542:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757687/real 1787757687] req@ffff8c05b2d98700 x1874588654974208/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757687.894543:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787757687.894549:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757687.894561:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10744 00000100:00080000:1.0:1787757687.894568:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757687.894578:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757687.894588:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757687.894606:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654974976/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757687.894623:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757687.894627:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757687.894630:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757687.894639:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10744 00000100:00080000:1.0:1787757687.894645:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787757687.894645:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654974208/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757687.894657:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757687.894666:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757687.894684:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588654975104/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757687.894689:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757687.894706:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757687.894716:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654974208:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757687.894730:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654974336:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757687.894770:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757687.894789:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757687/real 1787757687] req@ffff8c05b2d9a300 x1874588654974336/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757687.894805:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588654974336/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757687.894816:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757687.894820:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757687.894825:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654974336:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757687.894854:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654974464:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757687.894919:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757687.894936:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757687/real 1787757687] req@ffff8c05b2d99f80 x1874588654974464/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757687.894952:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654974464/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757687.894971:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757687.894975:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757687.894980:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654974464:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757687.894987:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654974592:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757687.895012:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757687.895024:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757687/real 1787757687] req@ffff8c05b2d98000 x1874588654974592/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757687.895038:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654974592/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757687.895048:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757687.895051:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757687.895055:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654974592:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757687.895061:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654974720:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757687.895080:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757687.895092:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757687/real 1787757687] req@ffff8c05b2d9b480 x1874588654974720/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757687.895105:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654974720/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757687.895115:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757687.895118:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757687.895123:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654974720:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757687.895130:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654974848:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757687.895149:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757687.895187:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757687/real 1787757687] req@ffff8c05b2d98380 x1874588654974848/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757687.895204:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654974848/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757687.895214:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757687.895218:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757687.895222:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654974848:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757687.895228:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654974976:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757687.895255:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757687.895271:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757687/real 1787757687] req@ffff8c05b2d99880 x1874588654974976/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757687.895295:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654974976/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757687.895311:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757687.895317:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757687.895324:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654974976:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757687.895336:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654975104:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757687.895376:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757687.895394:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757687/real 1787757687] req@ffff8c05b2d9b100 x1874588654975104/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757687.895410:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654975104/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757687.895421:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757687.895425:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757687.895429:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654975104:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757688.021993:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757688.470270:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757689.046023:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757689.494043:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757690.069998:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757690.518188:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757691.094031:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757691.542137:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757692.118025:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757692.566242:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757693.014002:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757693.014014:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787757693.014020:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757693.014032:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10749 00000100:00080000:1.0:1787757693.014045:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757693.014053:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757693.014136:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588654975232/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757693.014173:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757693.014178:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757693.014181:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757693.014186:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10749 00000100:00080000:1.0:1787757693.014191:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757693.014196:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757693.014207:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757693.014227:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588654975360/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757693.014245:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757693.014253:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757693.014257:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757693.014257:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654975232:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757693.014268:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10749 00000100:00080000:1.0:1787757693.014277:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757693.014281:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757693.014287:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757693.014303:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588654975488/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757693.014317:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757693.014321:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757693.014323:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757693.014328:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10749 00000100:00080000:1.0:1787757693.014333:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757693.014337:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757693.014344:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757693.014359:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654975616/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787757693.014366:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757693.014383:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757693.014388:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757693.014391:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757693.014399:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10749 00000100:00080000:1.0:1787757693.014404:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757693.014409:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757693.014416:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757693.014443:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654975744/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757693.014444:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757693/real 1787757693] req@ffff8c05b2d9aa00 x1874588654975232/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757693.014487:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757693.014495:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757693.014499:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757693.014507:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588654975232/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757693.014514:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10749 00000100:00080000:1.0:1787757693.014527:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757693.014535:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757693.014541:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:3.0:1787757693.014548:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757693.014558:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654975232:192.168.201.107@tcp:250:kworker.0 00000008:00100000:1.0:1787757693.014560:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757693.014593:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588654975872/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757693.014595:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654975360:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787757693.014629:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757693.014634:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787757693.014637:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787757693.014643:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757693.014651:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10749 00000100:00080000:1.0:1787757693.014660:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757693.014668:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757693.014675:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757693/real 1787757693] req@ffff8c05b2d9bb80 x1874588654975360/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787757693.014677:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757693.014707:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654976000/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757693.014720:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588654975360/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757693.014781:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757693.014783:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757693.014791:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757693.014798:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787757693.014800:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757693.014809:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654975360:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787757693.014815:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10749 00000100:00080000:1.0:1787757693.014826:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757693.014835:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757693.014840:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757693.014849:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654975488:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757693.014853:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654976128/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787757693.014933:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757693.014954:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757693/real 1787757693] req@ffff8c05b2d9b100 x1874588654975488/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757693.014972:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654975488/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757693.014983:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757693.014987:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757693.014991:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654975488:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757693.014999:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654975616:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757693.015021:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757693.015032:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757693/real 1787757693] req@ffff8c05b2d99880 x1874588654975616/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757693.015046:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654975616/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757693.015064:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757693.015100:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757693.015105:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654975616:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757693.015114:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654975744:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757693.015142:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757693.015157:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757693/real 1787757693] req@ffff8c05b2d98380 x1874588654975744/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757693.015175:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654975744/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757693.015192:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757693.015198:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757693.015204:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654975744:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757693.015240:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654975872:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757693.015283:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757693.015302:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757693/real 1787757693] req@ffff8c05b2d9b480 x1874588654975872/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757693.015317:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654975872/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757693.015328:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757693.015331:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757693.015336:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654975872:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757693.015344:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654976000:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757693.015367:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757693.015378:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757693/real 1787757693] req@ffff8c05b2d98000 x1874588654976000/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757693.015392:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654976000/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757693.015401:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757693.015405:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757693.015408:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654976000:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757693.015415:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654976128:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757693.015433:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757693.015444:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757693/real 1787757693] req@ffff8c05b2d99f80 x1874588654976128/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757693.015458:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654976128/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757693.015467:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757693.015470:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757693.015474:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654976128:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757693.142013:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757693.590178:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757694.166006:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757694.614285:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757695.190012:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757695.638125:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787757696.214050:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757696.662284:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757697.238067:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757697.686124:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757698.134048:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757698.134061:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757698.134066:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757698.134078:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10754 00000100:00080000:3.0:1787757698.134090:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757698.134099:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757698.134146:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588654976256/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757698.134185:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757698.134190:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757698.134193:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757698.134198:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10754 00000100:00080000:3.0:1787757698.134202:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757698.134207:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757698.134218:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757698.134234:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588654976384/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757698.134249:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757698.134254:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757698.134257:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757698.134262:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10754 00000100:00080000:3.0:1787757698.134266:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757698.134270:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757698.134276:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757698.134292:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654976256:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757698.134293:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588654976512/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757698.134323:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757698.134327:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757698.134329:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757698.134334:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10754 00000100:00080000:3.0:1787757698.134340:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757698.134344:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757698.134351:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757698.134366:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588654976640/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757698.134391:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757698.134396:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757698.134400:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757698.134405:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10754 00000100:00080000:3.0:1787757698.134410:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757698.134415:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:2.0:1787757698.134425:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:3.0:1787757698.134425:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757698.134449:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588654976768/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757698.134462:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757698.134466:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757698.134468:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757698.134476:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10754 00000100:00080000:3.0:1787757698.134483:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757698.134488:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757698.134495:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757698.134511:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757698/real 1787757698] req@ffff8c05a9f7b480 x1874588654976256/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757698.134523:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588654976896/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757698.134548:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787757698.134554:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588654976256/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757698.134561:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757698.134571:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757698.134584:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757698.134591:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10754 00000100:00080000:2.0:1787757698.134597:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787757698.134608:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787757698.134609:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654976256:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757698.134625:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757698.134633:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654976384:192.168.201.107@tcp:38:kworker.0 00000008:00100000:3.0:1787757698.134640:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757698.134671:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588654977024/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787757698.134687:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757698.134700:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757698.134707:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757698.134711:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787757698.134718:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757698/real 1787757698] req@ffff8c05a9f78380 x1874588654976384/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757698.134725:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10754 00000100:00080000:3.0:1787757698.134737:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757698.134750:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757698.134757:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757698.134758:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588654976384/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757698.134777:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787757698.134780:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588654977152/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757698.134788:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757698.134803:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654976384:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757698.134847:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654976512:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757698.134873:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757698.134932:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757698/real 1787757698] req@ffff8c05a9f78a80 x1874588654976512/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757698.134948:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588654976512/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757698.134958:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757698.134963:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757698.134967:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654976512:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757698.134975:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654976640:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757698.135036:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757698.135052:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757698/real 1787757698] req@ffff8c05a9f7a680 x1874588654976640/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757698.135067:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588654976640/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757698.135078:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757698.135082:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757698.135086:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654976640:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757698.135093:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654976768:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757698.135112:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757698.135124:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757698/real 1787757698] req@ffff8c05a9f7aa00 x1874588654976768/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757698.135138:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588654976768/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757698.135148:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757698.135151:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757698.135155:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654976768:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757698.135164:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654976896:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757698.135199:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757698.135222:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757698/real 1787757698] req@ffff8c05a9f7a300 x1874588654976896/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757698.135248:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588654976896/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757698.135264:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757698.135270:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757698.135279:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654976896:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757698.135292:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654977024:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757698.135335:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757698.135357:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757698/real 1787757698] req@ffff8c05a9f79880 x1874588654977024/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757698.135373:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588654977024/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757698.135383:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757698.135387:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757698.135391:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654977024:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757698.135399:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654977152:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757698.135424:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757698.135435:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757698/real 1787757698] req@ffff8c05a9f78700 x1874588654977152/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757698.135449:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588654977152/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757698.135459:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757698.135462:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757698.135466:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654977152:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757698.262054:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757698.710260:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757699.286078:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757699.734125:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757700.310032:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757700.758228:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757701.334058:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757701.782136:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757702.358042:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757702.806285:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757703.254050:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757703.254063:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757703.254069:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757703.254081:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10759 00000100:00080000:3.0:1787757703.254093:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757703.254103:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757703.254151:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588654977280/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757703.254188:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757703.254193:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757703.254195:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757703.254201:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10759 00000100:00080000:3.0:1787757703.254205:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757703.254209:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757703.254220:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757703.254264:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588654977408/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757703.254280:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757703.254286:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757703.254288:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757703.254295:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10759 00000100:00080000:3.0:1787757703.254301:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757703.254305:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757703.254311:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757703.254327:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588654977536/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757703.254349:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654977280:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757703.254353:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757703.254374:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757703.254378:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757703.254385:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10759 00000100:00080000:3.0:1787757703.254388:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757703.254392:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757703.254400:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757703.254424:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588654977664/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757703.254437:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757703.254442:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757703.254445:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757703.254451:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10759 00000100:00080000:3.0:1787757703.254478:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757703.254485:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757703.254494:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787757703.254506:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757703.254516:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588654977792/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757703.254540:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757703.254546:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757703.254551:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757703.254561:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10759 00000100:00080000:3.0:1787757703.254570:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757703.254579:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757703.254590:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757703.254611:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757703/real 1787757703] req@ffff8c05a9f78700 x1874588654977280/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757703.254642:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588654977920/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757703.254651:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588654977280/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757703.254730:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757703.254736:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757703.254738:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757703.254760:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654977280:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757703.254761:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757703.254772:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787757703.254784:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654977408:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757703.254784:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10759 00000100:00080000:3.0:1787757703.254805:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757703.254817:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757703.254827:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787757703.254841:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757703.254852:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588654978048/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757703.254868:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757703/real 1787757703] req@ffff8c05a9f79880 x1874588654977408/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757703.254959:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787757703.254964:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588654977408/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757703.254979:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757703.254992:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757703.254994:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757703.255018:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787757703.255024:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10759 00000100:00100000:2.0:1787757703.255030:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654977408:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757703.255046:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757703.255061:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757703.255071:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757703.255083:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654977536:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757703.255096:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588654978176/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787757703.255123:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757703.255149:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757703/real 1787757703] req@ffff8c05a9f7a300 x1874588654977536/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757703.255172:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588654977536/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757703.255183:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757703.255188:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757703.255192:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654977536:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757703.255201:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654977664:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757703.255234:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757703.255253:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757703/real 1787757703] req@ffff8c05a9f7aa00 x1874588654977664/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757703.255270:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588654977664/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757703.255286:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757703.255295:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757703.255299:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654977664:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757703.255307:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654977792:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757703.255332:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757703.255351:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757703/real 1787757703] req@ffff8c05a9f7a680 x1874588654977792/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757703.255378:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588654977792/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757703.255407:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757703.255416:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757703.255424:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654977792:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757703.255434:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654977920:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757703.255465:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757703.255483:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757703/real 1787757703] req@ffff8c05a9f78a80 x1874588654977920/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757703.255497:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588654977920/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757703.255508:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757703.255511:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757703.255518:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654977920:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757703.255527:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654978048:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757703.255556:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757703.255568:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757703/real 1787757703] req@ffff8c05a9f78380 x1874588654978048/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757703.255587:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588654978048/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757703.255598:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757703.255602:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757703.255607:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654978048:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757703.255642:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654978176:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757703.255665:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757703.255676:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757703/real 1787757703] req@ffff8c05a9f7b480 x1874588654978176/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757703.255689:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588654978176/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757703.255699:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757703.255703:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757703.255707:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654978176:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757703.381987:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757703.830225:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757704.406048:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757704.854247:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757705.429998:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757705.878125:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757706.453993:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757706.902257:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757707.477987:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757707.926116:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757708.374015:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757708.374028:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787757708.374034:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757708.374046:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10764 00000100:00080000:1.0:1787757708.374059:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757708.374068:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757708.374122:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588654978304/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757708.374174:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757708.374182:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757708.374186:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757708.374194:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10764 00000100:00080000:1.0:1787757708.374202:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757708.374210:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757708.374222:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757708.374257:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588654978432/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757708.374276:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757708.374284:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787757708.374288:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654978304:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757708.374289:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757708.374305:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10764 00000100:00080000:1.0:1787757708.374312:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757708.374317:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757708.374324:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757708.374341:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588654978560/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757708.374356:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757708.374361:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757708.374364:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757708.374369:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10764 00000100:00080000:1.0:1787757708.374375:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757708.374380:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757708.374388:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787757708.374392:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757708.374409:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588654978688/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757708.374424:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757708.374428:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757708.374431:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757708.374436:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10764 00000100:00080000:1.0:1787757708.374441:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757708.374447:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757708.374454:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757708.374456:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757708/real 1787757708] req@ffff8c05b2d9a300 x1874588654978304/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757708.374474:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588654978816/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757708.374491:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787757708.374492:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588654978304/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757708.374503:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757708.374509:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757708.374525:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10764 00000100:00080000:1.0:1787757708.374535:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757708.374541:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757708.374542:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757708.374554:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787757708.374563:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757708.374577:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654978304:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787757708.374589:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588654978944/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757708.374604:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757708.374608:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787757708.374611:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757708.374616:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10764 00000100:00080000:1.0:1787757708.374620:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757708.374631:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757708.374633:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654978432:192.168.201.107@tcp:38:kworker.0 00000008:00100000:1.0:1787757708.374638:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757708.374660:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588654979072/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757708.374676:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757708.374682:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757708.374687:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787757708.374688:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757708.374698:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10764 00000100:00080000:1.0:1787757708.374706:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757708.374712:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757708.374718:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757708.374723:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757708/real 1787757708] req@ffff8c05b2d98700 x1874588654978432/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757708.374740:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588654979200/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757708.374753:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588654978432/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757708.374773:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757708.374778:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757708.374787:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654978432:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757708.374828:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654978560:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757708.374926:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757708.374951:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757708/real 1787757708] req@ffff8c05b2d99f80 x1874588654978560/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757708.374969:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588654978560/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757708.374980:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757708.374984:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757708.374989:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654978560:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757708.374997:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654978688:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757708.375026:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757708.375039:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757708/real 1787757708] req@ffff8c05b2d98000 x1874588654978688/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757708.375053:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588654978688/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757708.375064:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757708.375067:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757708.375071:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654978688:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757708.375078:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654978816:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757708.375098:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757708.375109:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757708/real 1787757708] req@ffff8c05b2d9b480 x1874588654978816/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757708.375123:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588654978816/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757708.375134:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757708.375137:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757708.375140:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654978816:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757708.375147:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654978944:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757708.375165:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757708.375177:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757708/real 1787757708] req@ffff8c05b2d98380 x1874588654978944/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757708.375191:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588654978944/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757708.375200:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757708.375204:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757708.375211:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654978944:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757708.375248:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654979072:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757708.375292:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757708.375316:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757708/real 1787757708] req@ffff8c05b2d99880 x1874588654979072/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757708.375333:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588654979072/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757708.375344:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757708.375348:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757708.375352:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654979072:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757708.375360:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654979200:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757708.375386:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757708.375398:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757708/real 1787757708] req@ffff8c05b2d9b100 x1874588654979200/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757708.375412:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588654979200/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757708.375422:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757708.375425:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757708.375429:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654979200:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757708.502010:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757708.950312:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757709.526043:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757709.974137:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757710.550028:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757710.998307:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757711.573946:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757712.022006:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757712.597949:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757713.046244:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757713.494051:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757713.494065:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787757713.494072:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757713.494087:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10769 00000100:00080000:0.0:1787757713.494104:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757713.494115:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757713.494199:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588654979328/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757713.494241:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757713.494246:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757713.494249:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757713.494255:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10769 00000100:00080000:0.0:1787757713.494261:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757713.494268:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757713.494281:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757713.494301:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588654979456/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757713.494319:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654979328:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757713.494325:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757713.494343:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757713.494347:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757713.494354:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10769 00000100:00080000:0.0:1787757713.494359:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757713.494365:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757713.494371:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757713.494386:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588654979584/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757713.494400:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757713.494405:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757713.494407:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757713.494415:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10769 00000100:00080000:0.0:1787757713.494420:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757713.494424:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757713.494433:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757713.494447:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588654979712/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787757713.494465:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757713.494468:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757713.494481:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757713.494485:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757713.494495:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10769 00000100:00080000:0.0:1787757713.494503:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757713.494509:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757713.494516:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757713.494528:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588654979840/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757713.494548:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757713.494554:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757713.494558:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757713.494568:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10769 00000100:00100000:2.0:1787757713.494575:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757713/real 1787757713] req@ffff8c05a9f74380 x1874588654979328/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757713.494580:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757713.494606:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757713.494620:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757713.494622:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588654979328/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757713.494661:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787757713.494664:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588654979968/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757713.494691:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787757713.494700:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787757713.494730:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654979328:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757713.494733:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787757713.494744:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757713.494767:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10769 00000100:00080000:0.0:1787757713.494776:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757713.494785:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757713.494797:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757713.494808:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654979456:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757713.494832:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588654980096/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787757713.494857:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757713.494863:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757713.494870:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757713.494873:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757713.494940:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10769 00000100:00100000:2.0:1787757713.494949:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757713/real 1787757713] req@ffff8c05a9f74a80 x1874588654979456/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757713.494958:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757713.494981:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757713.494995:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757713.495003:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588654979456/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757713.495024:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588654980224/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757713.495035:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757713.495053:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757713.495064:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654979456:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757713.495082:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654979584:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757713.495129:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757713.495153:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757713/real 1787757713] req@ffff8c05a9f75c00 x1874588654979584/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757713.495172:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588654979584/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757713.495192:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757713.495197:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757713.495202:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654979584:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757713.495210:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654979712:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757713.495235:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757713.495247:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757713/real 1787757713] req@ffff8c05a9f77800 x1874588654979712/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757713.495262:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588654979712/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757713.495272:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757713.495276:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757713.495279:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654979712:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757713.495286:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654979840:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757713.495307:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757713.495319:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757713/real 1787757713] req@ffff8c05a9f76680 x1874588654979840/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757713.495333:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588654979840/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757713.495343:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757713.495346:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757713.495351:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654979840:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757713.495358:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654979968:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757713.495379:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757713.495389:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757713/real 1787757713] req@ffff8c05a9f77b80 x1874588654979968/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757713.495403:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588654979968/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757713.495413:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757713.495417:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757713.495421:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654979968:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757713.495448:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654980096:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757713.495469:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757713.495481:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757713/real 1787757713] req@ffff8c05a9f76a00 x1874588654980096/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757713.495494:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588654980096/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757713.495504:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757713.495508:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757713.495511:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654980096:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757713.495518:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654980224:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757713.495536:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757713.495548:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757713/real 1787757713] req@ffff8c05a9f74380 x1874588654980224/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757713.495562:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588654980224/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757713.495572:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757713.495576:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757713.495579:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654980224:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757713.622029:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757714.070192:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757714.646029:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757715.094354:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757715.670019:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757716.118211:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757716.693999:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757717.142112:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757717.718047:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757718.166231:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757718.614080:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757718.614093:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757718.614099:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757718.614112:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10775 00000100:00080000:3.0:1787757718.614125:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757718.614135:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757718.614181:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588654980352/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757718.614221:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757718.614226:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757718.614229:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757718.614234:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10775 00000100:00080000:3.0:1787757718.614238:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757718.614243:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757718.614254:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757718.614269:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588654980480/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757718.614282:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757718.614286:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757718.614288:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757718.614292:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10775 00000100:00080000:3.0:1787757718.614296:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757718.614299:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757718.614305:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757718.614316:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588654980608/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757718.614338:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757718.614344:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757718.614346:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757718.614352:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10775 00000100:00080000:3.0:1787757718.614357:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757718.614362:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757718.614368:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757718.614381:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588654980736/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757718.614395:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757718.614399:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757718.614404:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757718.614414:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10775 00000100:00100000:2.0:1787757718.614416:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654980352:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757718.614420:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757718.614442:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757718.614449:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757718.614473:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588654980864/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757718.614485:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757718.614489:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757718.614492:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757718.614499:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10775 00000100:00080000:3.0:1787757718.614504:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757718.614510:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757718.614517:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757718.614529:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588654980992/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757718.614558:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787757718.614573:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757718.614575:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757718.614582:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757718.614590:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10775 00000100:00080000:3.0:1787757718.614596:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757718.614602:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757718.614612:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757718.614657:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588654981120/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757718.614671:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757718.614676:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757718.614679:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757718.614688:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10775 00000100:00100000:2.0:1787757718.614698:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757718/real 1787757718] req@ffff8c05a9f7ad80 x1874588654980352/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757718.614699:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757718.614721:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757718.614741:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757718.614769:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588654981248/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757718.614771:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588654980352/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757718.614951:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757718.614962:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757718.614972:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654980352:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787757718.614987:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654980480:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757718.615023:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757718.615041:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757718/real 1787757718] req@ffff8c05a9f7b800 x1874588654980480/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757718.615059:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588654980480/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757718.615079:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757718.615083:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757718.615088:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654980480:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757718.615095:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654980608:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757718.615116:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757718.615128:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757718/real 1787757718] req@ffff8c05a9f78e00 x1874588654980608/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757718.615142:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588654980608/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757718.615152:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757718.615156:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757718.615160:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654980608:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757718.615200:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654980736:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757718.615222:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757718.615235:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757718/real 1787757718] req@ffff8c05a9f79500 x1874588654980736/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757718.615249:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588654980736/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757718.615259:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757718.615263:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757718.615267:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654980736:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757718.615275:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654980864:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757718.615295:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757718.615307:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757718/real 1787757718] req@ffff8c05a9f7b100 x1874588654980864/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757718.615321:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588654980864/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757718.615331:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757718.615334:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757718.615338:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654980864:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757718.615345:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654980992:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757718.615363:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757718.615374:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757718/real 1787757718] req@ffff8c05a9f79180 x1874588654980992/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757718.615388:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588654980992/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757718.615398:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757718.615402:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757718.615406:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654980992:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757718.615413:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654981120:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757718.615432:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757718.615444:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757718/real 1787757718] req@ffff8c05a9f79f80 x1874588654981120/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757718.615458:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588654981120/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757718.615468:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757718.615472:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757718.615475:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654981120:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757718.615482:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654981248:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757718.615501:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757718.615512:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757718/real 1787757718] req@ffff8c05a9f7bb80 x1874588654981248/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757718.615526:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588654981248/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757718.615536:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757718.615540:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757718.615543:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654981248:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757718.741992:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757719.190252:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757719.765987:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757720.214134:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757720.790019:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757721.238255:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757721.813979:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757722.262162:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757722.837993:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757723.286329:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757723.733969:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757723.733982:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757723.733988:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757723.734002:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10780 00000100:00080000:3.0:1787757723.734016:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757723.734042:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757723.734108:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588654981376/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757723.734157:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757723.734165:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757723.734169:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757723.734178:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10780 00000100:00080000:3.0:1787757723.734186:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757723.734195:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757723.734211:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757723.734237:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588654981504/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757723.734260:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757723.734269:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757723.734274:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757723.734284:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10780 00000100:00080000:3.0:1787757723.734295:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757723.734304:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757723.734318:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757723.734341:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588654981632/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757723.734350:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654981376:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757723.734369:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757723.734375:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757723.734378:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757723.734386:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10780 00000100:00080000:3.0:1787757723.734393:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757723.734401:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757723.734409:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757723.734426:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588654981760/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757723.734439:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757723.734445:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757723.734448:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757723.734452:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10780 00000100:00080000:3.0:1787757723.734473:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757723.734479:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757723.734489:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757723.734510:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588654981888/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787757723.734519:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757723.734570:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757723.734577:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757723.734582:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757723.734596:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10780 00000100:00080000:3.0:1787757723.734606:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757723.734617:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757723.734630:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757723.734651:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757723/real 1787757723] req@ffff8c05a9f7bb80 x1874588654981376/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757723.734662:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588654982016/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757723.734741:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757723.734746:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787757723.734755:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588654981376/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757723.734758:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757723.734789:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10780 00000100:00080000:2.0:1787757723.734792:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757723.734810:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787757723.734810:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787757723.734822:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654981376:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757723.734833:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757723.734845:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654981504:192.168.201.107@tcp:38:kworker.0 00000008:00100000:3.0:1787757723.734851:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757723.734938:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588654982144/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757723.734954:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757723.734959:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757723.734966:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757723.734976:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10780 00000100:00080000:3.0:1787757723.734986:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787757723.735001:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757723.735008:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757723.735021:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757723.735031:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757723/real 1787757723] req@ffff8c05a9f79f80 x1874588654981504/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757723.735043:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588654982272/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757723.735106:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588654981504/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757723.735120:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757723.735130:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757723.735136:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654981504:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757723.735178:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654981632:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757723.735205:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757723.735219:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757723/real 1787757723] req@ffff8c05a9f79180 x1874588654981632/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757723.735234:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588654981632/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757723.735245:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757723.735249:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757723.735253:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654981632:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757723.735262:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654981760:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757723.735284:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757723.735296:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757723/real 1787757723] req@ffff8c05a9f7b100 x1874588654981760/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757723.735310:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588654981760/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757723.735321:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757723.735325:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757723.735328:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654981760:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757723.735335:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654981888:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757723.735354:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757723.735365:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757723/real 1787757723] req@ffff8c05a9f79500 x1874588654981888/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757723.735379:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588654981888/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757723.735390:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757723.735393:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757723.735397:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654981888:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757723.735403:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654982016:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757723.735422:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757723.735433:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757723/real 1787757723] req@ffff8c05a9f78e00 x1874588654982016/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757723.735448:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588654982016/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757723.735458:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757723.735462:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757723.735465:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654982016:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757723.735472:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654982144:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757723.735491:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757723.735502:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757723/real 1787757723] req@ffff8c05a9f7b800 x1874588654982144/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757723.735515:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588654982144/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757723.735546:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757723.735551:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757723.735556:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654982144:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757723.735564:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654982272:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757723.735588:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757723.735602:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757723/real 1787757723] req@ffff8c05a9f7ad80 x1874588654982272/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757723.735616:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588654982272/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757723.735636:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757723.735640:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757723.735644:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654982272:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757723.862019:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757724.310155:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757724.885962:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757725.334138:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757725.910105:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757726.358076:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757726.934007:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757727.382268:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757727.958097:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757728.406245:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757728.854073:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757728.854086:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787757728.854091:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757728.854102:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10785 00000100:00080000:2.0:1787757728.854116:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757728.854125:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757728.854172:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73800 x1874588654982400/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757728.854212:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757728.854216:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757728.854219:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757728.854224:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10785 00000100:00080000:2.0:1787757728.854229:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757728.854234:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757728.854246:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757728.854264:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588654982528/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757728.854277:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654982400:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757728.854286:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757728.854292:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757728.854295:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757728.854302:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10785 00000100:00080000:2.0:1787757728.854307:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757728.854311:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757728.854318:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757728.854336:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588654982656/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757728.854350:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757728.854356:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757728.854361:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757728.854369:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10785 00000800:00000100:3.0:1787757728.854377:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757728.854378:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757728.854396:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757728.854408:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757728.854460:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588654982784/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757728.854481:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757728/real 1787757728] req@ffff8c05b2e73800 x1874588654982400/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757728.854500:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757728.854510:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1787757728.854515:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73800 x1874588654982400/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757728.854518:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757728.854539:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10785 00000100:00080000:3.0:1787757728.854543:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757728.854552:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757728.854553:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757728.854568:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654982400:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757728.854575:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757728.854590:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757728.854605:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654982528:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757728.854621:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588654982912/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757728.854640:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757728.854645:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757728.854651:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787757728.854653:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757728.854661:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10785 00000100:00080000:2.0:1787757728.854670:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757728.854678:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757728.854684:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757728/real 1787757728] req@ffff8c05b2e70380 x1874588654982528/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787757728.854689:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757728.854720:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588654983040/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757728.854721:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588654982528/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757728.854785:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757728.854788:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757728.854797:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787757728.854799:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787757728.854808:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757728.854813:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654982528:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757728.854819:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10785 00000100:00080000:2.0:1787757728.854829:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787757728.854834:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654982656:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757728.854843:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757728.854853:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787757728.854905:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757728.854927:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757728/real 1787757728] req@ffff8c05b2e71880 x1874588654982656/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757728.854945:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588654982656/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757728.854954:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588654983168/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757728.854974:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757728.854986:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757728.854988:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787757728.855000:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787757728.855001:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654982656:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757728.855007:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757728.855020:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654982784:192.168.201.107@tcp:8:kworker.0 00000100:00080000:2.0:1787757728.855021:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10785 00000100:00080000:2.0:1787757728.855036:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757728.855045:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757728.855053:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757728.855069:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588654983296/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787757728.855084:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757728.855109:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757728/real 1787757728] req@ffff8c05b2e71c00 x1874588654982784/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757728.855134:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588654982784/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757728.855146:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757728.855149:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757728.855153:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654982784:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757728.855173:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654982912:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757728.855195:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757728.855207:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757728/real 1787757728] req@ffff8c05b2e73b80 x1874588654982912/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757728.855221:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588654982912/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757728.855231:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757728.855234:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757728.855238:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654982912:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757728.855245:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654983040:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757728.855263:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757728.855273:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757728/real 1787757728] req@ffff8c05b2e71500 x1874588654983040/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757728.855287:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588654983040/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757728.855305:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757728.855309:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757728.855312:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654983040:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757728.855319:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654983168:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757728.855337:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757728.855347:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757728/real 1787757728] req@ffff8c05b2e73480 x1874588654983168/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757728.855361:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588654983168/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757728.855371:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757728.855374:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757728.855378:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654983168:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757728.855384:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654983296:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757728.855402:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757728.855412:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757728/real 1787757728] req@ffff8c05b2e70700 x1874588654983296/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757728.855425:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588654983296/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757728.855456:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757728.855461:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757728.855466:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654983296:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757728.982010:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757729.430327:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757730.006011:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757730.454103:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757731.030047:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757731.478245:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757732.053998:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757732.502155:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757733.078122:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757733.526198:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757733.973989:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757733.974002:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787757733.974007:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757733.974019:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10790 00000100:00080000:2.0:1787757733.974032:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757733.974041:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757733.974086:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588654983424/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757733.974123:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757733.974128:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757733.974131:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757733.974136:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10790 00000100:00080000:2.0:1787757733.974141:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757733.974145:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757733.974155:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757733.974170:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588654983552/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757733.974182:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757733.974187:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757733.974190:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757733.974194:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10790 00000100:00080000:2.0:1787757733.974198:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757733.974203:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757733.974208:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757733.974221:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588654983680/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757733.974233:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757733.974237:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757733.974239:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757733.974244:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10790 00000100:00080000:2.0:1787757733.974247:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757733.974252:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757733.974260:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757733.974264:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654983424:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787757733.974277:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588654983808/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757733.974292:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757733.974296:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757733.974298:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757733.974302:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10790 00000100:00080000:2.0:1787757733.974306:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757733.974311:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757733.974317:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757733.974333:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588654983936/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757733.974380:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757733.974384:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757733.974386:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757733.974391:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10790 00000100:00080000:2.0:1787757733.974396:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757733.974401:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757733.974407:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757733.974432:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588654984064/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757733.974444:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787757733.974453:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654983552:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757733.974454:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787757733.974462:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757733.974476:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10790 00000100:00080000:2.0:1787757733.974482:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757733.974488:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757733.974496:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757733.974516:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588654984192/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757733.974521:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654983680:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757733.974549:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757733.974554:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757733.974559:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757733.974566:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10790 00000100:00080000:2.0:1787757733.974572:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757733.974576:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757733.974583:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757733.974590:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654983808:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757733.974603:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588654984320/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757733.974632:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654983936:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757733.974661:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654984064:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757733.974687:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654984192:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757733.974715:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654984320:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757734.101966:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757734.550099:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757735.125991:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757735.574241:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757736.149986:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757736.598118:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787757737.046354:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.107:988 00000400:00000100:1.0:1787757737.046429:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.107@tcp at host 192.168.201.107 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787757737.046457:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787757737.046523:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787757737.046534:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787757737.046545:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787757737.046558:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787757737.046573:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787757737.046589:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787757737.046603:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:3.0:1787757737.046622:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757733/real 1787757737] req@ffff8c05b2e70e00 x1874588654983808/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757737.046694:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588654983808/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757737.046746:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757737.046757:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757737.046765:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654983808:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757737.046783:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757733/real 1787757737] req@ffff8c05b2e70000 x1874588654983936/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757737.046810:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588654983936/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757737.046826:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757737.046833:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757737.046839:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654983936:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757737.046852:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757733/real 1787757737] req@ffff8c05b2e72300 x1874588654984064/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757737.046930:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588654984064/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757737.046950:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757737.046959:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757737.046967:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654984064:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757737.046982:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757733/real 1787757737] req@ffff8c05b2e72a00 x1874588654984192/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757737.047010:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588654984192/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757737.047028:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757737.047034:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757737.047041:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654984192:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757737.047055:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757733/real 1787757737] req@ffff8c05b2e73100 x1874588654983424/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757737.047079:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588654983424/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757737.047094:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757737.047100:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757737.047107:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654983424:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757737.047119:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757733/real 1787757737] req@ffff8c05b2e72d80 x1874588654983552/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757737.047140:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588654983552/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757737.047155:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757737.047161:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757737.047166:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654983552:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757737.047178:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757733/real 1787757737] req@ffff8c05b2e72680 x1874588654983680/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757737.047201:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588654983680/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757737.047229:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757737.047237:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757737.047244:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654983680:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757737.047704:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757733/real 1787757737] req@ffff8c05b2e71180 x1874588654984320/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757737.047734:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588654984320/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757737.047750:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757737.047757:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757737.047763:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654984320:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757737.174012:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757737.622164:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757738.198010:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757738.646114:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757739.094034:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757739.094046:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787757739.094052:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757739.094065:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10795 00000100:00080000:2.0:1787757739.094078:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757739.094087:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757739.094133:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588654984448/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757739.094176:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757739.094181:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757739.094184:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757739.094189:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10795 00000100:00080000:2.0:1787757739.094193:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757739.094198:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757739.094208:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757739.094224:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588654984576/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757739.094246:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757739.094250:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757739.094252:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757739.094290:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10795 00000100:00080000:2.0:1787757739.094299:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757739.094308:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757739.094319:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757739.094335:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654984448:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787757739.094347:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588654984704/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757739.094379:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757739.094386:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757739.094389:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757739.094397:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10795 00000100:00080000:2.0:1787757739.094406:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757739.094414:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757739.094437:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757739.094465:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588654984832/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787757739.094475:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757739.094500:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757739.094507:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757739.094510:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757739.094517:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10795 00000100:00080000:2.0:1787757739.094525:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757739.094531:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757739.094539:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757739.094561:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588654984960/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757739.094573:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757739/real 1787757739] req@ffff8c05b2e70a80 x1874588654984448/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757739.094579:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757739.094595:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757739.094603:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757739.094620:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10795 00000100:00080000:2.0:1787757739.094632:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787757739.094640:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588654984448/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757739.094648:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757739.094663:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757739.094691:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588654985088/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757739.094706:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757739.094720:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757739.094723:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787757739.094737:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787757739.094744:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654984448:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757739.094746:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757739.094765:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10795 00000100:00100000:3.0:1787757739.094774:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654984576:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757739.094777:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757739.094790:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757739.094809:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757739.094838:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588654985216/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757739.094871:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787757739.094932:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757739.094956:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757739.094961:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757739.094972:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757739/real 1787757739] req@ffff8c05b2e71180 x1874588654984576/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757739.094975:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10795 00000100:00080000:2.0:1787757739.094990:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757739.095004:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757739.095018:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757739.095020:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588654984576/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757739.095048:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787757739.095052:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588654985344/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757739.095062:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757739.095079:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654984576:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757739.095707:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654984704:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757739.095763:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757739.095791:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757739/real 1787757739] req@ffff8c05b2e72680 x1874588654984704/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757739.095818:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588654984704/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757739.095835:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757739.095842:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757739.095849:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654984704:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757739.095863:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654984832:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757739.095941:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757739.095966:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757739/real 1787757739] req@ffff8c05b2e72d80 x1874588654984832/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757739.095992:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588654984832/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757739.096012:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757739.096020:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757739.096028:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654984832:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757739.096043:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654984960:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757739.096185:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757739.096211:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757739/real 1787757739] req@ffff8c05b2e73100 x1874588654984960/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757739.096230:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588654984960/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757739.096242:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757739.096247:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757739.096251:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654984960:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757739.096289:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654985088:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757739.096331:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757739.096356:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757739/real 1787757739] req@ffff8c05b2e72a00 x1874588654985088/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757739.096381:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588654985088/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757739.096398:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757739.096404:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757739.096410:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654985088:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757739.096420:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654985216:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757739.096468:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757739.096487:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757739/real 1787757739] req@ffff8c05b2e72300 x1874588654985216/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757739.096521:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588654985216/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757739.096533:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757739.096537:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757739.096541:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654985216:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757739.096547:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654985344:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757739.096567:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757739.096577:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757739/real 1787757739] req@ffff8c05b2e70000 x1874588654985344/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757739.096590:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588654985344/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757739.096609:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757739.096613:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757739.096616:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654985344:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757739.221997:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757739.670315:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757740.246000:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757740.694139:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757741.270006:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757741.718173:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757742.293998:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757742.742129:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757743.317991:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757743.766204:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757744.214010:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757744.214023:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787757744.214029:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757744.214043:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10800 00000100:00080000:2.0:1787757744.214057:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757744.214067:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757744.214120:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588654985472/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757744.214194:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757744.214201:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757744.214206:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757744.214217:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10800 00000100:00080000:2.0:1787757744.214226:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757744.214236:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757744.214248:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654985472:192.168.201.107@tcp:250:kworker.0 00000008:00100000:2.0:1787757744.214257:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757744.214287:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588654985600/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757744.214306:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757744.214311:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757744.214314:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757744.214319:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10800 00000100:00080000:2.0:1787757744.214326:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757744.214331:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757744.214338:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787757744.214344:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757744.214359:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588654985728/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757744.214375:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757744.214380:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757744.214382:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757744.214388:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10800 00000100:00080000:2.0:1787757744.214394:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757744.214398:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757744.214404:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757744.214420:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757744/real 1787757744] req@ffff8c05b2e70e00 x1874588654985472/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757744.214422:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588654985856/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757744.214488:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757744.214493:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757744.214497:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757744.214503:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588654985472/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757744.214508:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10800 00000100:00080000:2.0:1787757744.214517:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757744.214528:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:3.0:1787757744.214539:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787757744.214540:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787757744.214555:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757744.214566:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588654985984/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757744.214568:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654985472:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757744.214593:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757744.214598:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757744.214600:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757744.214606:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10800 00000100:00080000:2.0:1787757744.214611:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787757744.214617:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654985600:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757744.214622:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757744.214634:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757744.214655:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588654986112/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787757744.214670:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757744.214671:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757744.214681:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787757744.214685:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757744.214693:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10800 00000100:00080000:2.0:1787757744.214698:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787757744.214703:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757744/real 1787757744] req@ffff8c05b2e70700 x1874588654985600/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757744.214709:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757744.214721:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757744.214751:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588654985600/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757744.214756:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588654986240/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757744.214777:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757744.214792:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757744.214799:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654985600:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757744.214801:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787757744.214815:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654985728:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757744.214816:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757744.214827:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757744.214838:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10800 00000100:00080000:2.0:1787757744.214849:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787757744.214857:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757744.214858:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757744.214872:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757744.214919:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757744/real 1787757744] req@ffff8c05b2e73480 x1874588654985728/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757744.214938:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588654985728/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757744.214947:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588654986368/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757744.214968:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757744.214976:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757744.214981:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654985728:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757744.214993:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654985856:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757744.215023:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757744.215037:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757744/real 1787757744] req@ffff8c05b2e71500 x1874588654985856/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757744.215051:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588654985856/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757744.215070:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757744.215074:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757744.215078:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654985856:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757744.215084:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654985984:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757744.215104:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757744.215115:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757744/real 1787757744] req@ffff8c05b2e73b80 x1874588654985984/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757744.215129:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588654985984/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757744.215139:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757744.215142:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757744.215146:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654985984:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757744.215189:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654986112:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757744.215215:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757744.215229:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757744/real 1787757744] req@ffff8c05b2e71c00 x1874588654986112/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757744.215243:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588654986112/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757744.215254:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757744.215257:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757744.215262:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654986112:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757744.215270:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654986240:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757744.215289:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757744.215300:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757744/real 1787757744] req@ffff8c05b2e71880 x1874588654986240/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757744.215313:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588654986240/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757744.215323:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757744.215326:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757744.215330:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654986240:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757744.215336:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654986368:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757744.215354:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757744.215365:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757744/real 1787757744] req@ffff8c05b2e70380 x1874588654986368/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757744.215378:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588654986368/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757744.215388:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757744.215391:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757744.215395:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654986368:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757744.341983:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757744.790092:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757745.365980:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757745.814253:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757746.389976:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757746.838155:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757747.414021:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757747.862197:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757748.437986:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757748.886162:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757749.334061:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757749.334074:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757749.334107:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757749.334132:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10805 00000100:00080000:3.0:1787757749.334146:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757749.334155:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757749.334202:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588654986496/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757749.334242:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757749.334247:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757749.334250:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757749.334256:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10805 00000100:00080000:3.0:1787757749.334260:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757749.334266:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757749.334279:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757749.334298:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588654986624/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757749.334308:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654986496:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757749.334319:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757749.334325:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757749.334328:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757749.334335:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10805 00000100:00080000:3.0:1787757749.334340:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757749.334345:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757749.334352:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757749.334371:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588654986752/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757749.334386:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757749.334391:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757749.334393:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757749.334397:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10805 00000100:00080000:3.0:1787757749.334402:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787757749.334409:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757749.334416:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757749.334426:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757749.334443:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588654986880/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757749.334455:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757749.334460:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757749.334463:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757749.334469:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10805 00000100:00080000:3.0:1787757749.334473:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787757749.334477:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757749/real 1787757749] req@ffff8c05a9f7b480 x1874588654986496/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757749.334484:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757749.334496:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757749.334523:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588654987008/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757749.334531:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588654986496/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757749.334544:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757749.334557:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757749.334562:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757749.334575:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10805 00000100:00080000:2.0:1787757749.334580:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757749.334581:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757749.334594:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:2.0:1787757749.334602:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000008:00100000:3.0:1787757749.334603:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757749.334623:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654986496:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757749.334658:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588654987136/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757749.334666:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654986624:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757749.334692:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757749.334697:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757749.334700:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757749.334706:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10805 00000100:00080000:3.0:1787757749.334711:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757749.334716:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757749.334724:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787757749.334742:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757749.334748:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588654987264/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757749.334765:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787757749.334775:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757749/real 1787757749] req@ffff8c05a9f78380 x1874588654986624/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757749.334775:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757749.334782:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757749.334797:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10805 00000100:00080000:3.0:1787757749.334810:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757749.334823:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757749.334824:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588654986624/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787757749.334830:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787757749.334850:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787757749.334855:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588654987392/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757749.334861:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757749.334922:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654986624:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757749.334944:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654986752:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757749.334992:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757749.335016:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757749/real 1787757749] req@ffff8c05a9f78a80 x1874588654986752/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757749.335034:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588654986752/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757749.335045:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757749.335049:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757749.335054:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654986752:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757749.335062:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654986880:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757749.335121:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757749.335137:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757749/real 1787757749] req@ffff8c05a9f7a680 x1874588654986880/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757749.335164:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588654986880/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757749.335176:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757749.335180:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757749.335184:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654986880:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757749.335192:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654987008:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757749.335216:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757749.335228:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757749/real 1787757749] req@ffff8c05a9f7aa00 x1874588654987008/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757749.335242:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588654987008/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757749.335252:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757749.335255:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757749.335259:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654987008:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757749.335281:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654987136:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757749.335302:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757749.335315:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757749/real 1787757749] req@ffff8c05a9f7a300 x1874588654987136/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757749.335329:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588654987136/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757749.335339:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757749.335342:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757749.335346:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654987136:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757749.335353:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654987264:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757749.335371:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757749.335382:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757749/real 1787757749] req@ffff8c05a9f79880 x1874588654987264/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757749.335396:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588654987264/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757749.335405:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757749.335408:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757749.335412:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654987264:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757749.335418:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654987392:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757749.335436:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757749.335447:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757749/real 1787757749] req@ffff8c05a9f78700 x1874588654987392/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757749.335461:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588654987392/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757749.335471:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757749.335474:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757749.335478:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654987392:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757749.461982:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757749.910242:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757750.485988:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757750.934149:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757751.509994:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757751.958271:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757752.534019:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757752.982127:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757753.558001:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757754.006189:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757754.454077:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757754.454095:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787757754.454100:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757754.454113:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10810 00000100:00080000:0.0:1787757754.454126:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757754.454135:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757754.454181:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588654987520/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757754.454221:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757754.454226:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757754.454228:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757754.454234:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10810 00000100:00080000:0.0:1787757754.454239:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757754.454244:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757754.454256:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757754.454272:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588654987648/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757754.454292:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757754.454299:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757754.454303:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757754.454314:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10810 00000100:00100000:2.0:1787757754.454321:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654987520:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757754.454324:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757754.454341:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757754.454353:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757754.454381:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588654987776/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757754.454399:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757754.454405:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757754.454408:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757754.454413:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10810 00000100:00080000:0.0:1787757754.454420:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787757754.454424:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757754.454431:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757754.454444:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757754.454462:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588654987904/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757754.454476:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757754.454482:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757754.454485:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757754.454493:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10810 00000100:00100000:2.0:1787757754.454494:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757754/real 1787757754] req@ffff8c05a9f76d80 x1874588654987520/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757754.454497:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757754.454511:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757754.454525:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757754.454544:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588654987520/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757754.454549:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588654988032/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757754.454592:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757754.454593:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757754.454607:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757754.454610:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757754.454615:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787757754.454620:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10810 00000100:00080000:0.0:1787757754.454630:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787757754.454639:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654987520:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757754.454640:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757754.454651:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757754.454668:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588654988160/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757754.454675:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654987648:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757754.454690:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757754.454698:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787757754.454703:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757754.454713:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10810 00000100:00080000:0.0:1787757754.454723:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757754.454733:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:2.0:1787757754.454741:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:0.0:1787757754.454751:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757754.454768:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757754/real 1787757754] req@ffff8c05a9f76300 x1874588654987648/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757754.454786:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588654988288/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757754.454797:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588654987648/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757754.454838:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757754.454840:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757754.454854:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757754.454856:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787757754.454861:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787757754.454868:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654987648:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757754.454923:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10810 00000100:00080000:0.0:1787757754.454933:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757754.454947:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757754.454956:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757754.454976:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588654988416/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757754.454982:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654987776:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757754.455064:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757754.455087:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757754/real 1787757754] req@ffff8c05a9f75180 x1874588654987776/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757754.455103:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588654987776/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757754.455123:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757754.455128:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757754.455132:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654987776:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757754.455140:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654987904:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757754.455162:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757754.455174:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757754/real 1787757754] req@ffff8c05a9f77100 x1874588654987904/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757754.455188:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588654987904/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757754.455199:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757754.455203:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757754.455206:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654987904:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757754.455213:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654988032:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757754.455233:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757754.455253:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757754/real 1787757754] req@ffff8c05a9f75500 x1874588654988032/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757754.455276:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588654988032/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757754.455293:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757754.455300:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757754.455308:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654988032:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757754.455321:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654988160:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757754.455358:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757754.455374:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757754/real 1787757754] req@ffff8c05a9f75880 x1874588654988160/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757754.455389:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588654988160/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757754.455400:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757754.455403:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757754.455407:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654988160:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757754.455415:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654988288:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757754.455436:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757754.455447:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757754/real 1787757754] req@ffff8c05a9f74000 x1874588654988288/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757754.455461:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588654988288/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757754.455471:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757754.455474:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757754.455477:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654988288:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757754.455517:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654988416:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757754.455542:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757754.455555:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757754/real 1787757754] req@ffff8c05a9f74e00 x1874588654988416/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757754.455577:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588654988416/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757754.455588:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757754.455592:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757754.455596:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654988416:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757754.582008:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757755.030141:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757755.606056:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757756.054235:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757756.630037:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757757.078229:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757757.654064:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757758.102265:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757758.678051:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757759.126209:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757759.574126:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757759.574143:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757759.574150:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757759.574165:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10815 00000100:00080000:3.0:1787757759.574183:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757759.574195:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757759.574261:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588654988544/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757759.574312:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757759.574319:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757759.574321:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757759.574327:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10815 00000100:00080000:3.0:1787757759.574332:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757759.574337:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757759.574352:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757759.574369:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588654988672/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757759.574384:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757759.574390:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757759.574392:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757759.574397:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10815 00000100:00080000:3.0:1787757759.574401:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757759.574406:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757759.574412:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757759.574424:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588654988800/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757759.574440:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787757759.574441:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654988544:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757759.574453:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757759.574460:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757759.574466:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10815 00000100:00080000:3.0:1787757759.574469:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757759.574474:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757759.574482:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757759.574499:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588654988928/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757759.574512:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757759.574517:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757759.574520:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757759.574524:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10815 00000100:00080000:3.0:1787757759.574561:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757759.574567:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757759.574573:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757759.574591:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588654989056/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787757759.574600:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757759.574630:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757759.574638:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757759.574642:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757759.574653:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10815 00000100:00080000:3.0:1787757759.574661:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757759.574670:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757759.574681:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757759.574704:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757759/real 1787757759] req@ffff8c05a9f78700 x1874588654988544/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757759.574714:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588654989184/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757759.574757:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757759.574762:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757759.574766:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757759.574777:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10815 00000100:00100000:2.0:1787757759.574778:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588654988544/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757759.574783:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757759.574799:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757759.574815:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787757759.574825:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757759.574839:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757759.574842:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588654989312/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757759.574854:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654988544:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787757759.574960:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654988672:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757759.574990:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757759.574997:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757759.575001:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787757759.575016:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757759.575018:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10815 00000100:00080000:3.0:1787757759.575036:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757759.575048:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757759.575058:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757759/real 1787757759] req@ffff8c05a9f79880 x1874588654988672/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787757759.575066:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757759.575087:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588654988672/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757759.575111:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588654989440/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757759.575115:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757759.575132:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757759.575146:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654988672:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757759.575194:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654988800:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757759.575222:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757759.575236:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757759/real 1787757759] req@ffff8c05a9f7a300 x1874588654988800/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757759.575252:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588654988800/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757759.575263:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757759.575267:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757759.575271:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654988800:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757759.575278:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654988928:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757759.575299:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757759.575310:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757759/real 1787757759] req@ffff8c05a9f7aa00 x1874588654988928/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757759.575334:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588654988928/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757759.575351:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757759.575357:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757759.575366:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654988928:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757759.575379:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654989056:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757759.575421:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757759.575442:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757759/real 1787757759] req@ffff8c05a9f7a680 x1874588654989056/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757759.575458:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588654989056/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757759.575469:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757759.575473:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757759.575477:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654989056:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757759.575484:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654989184:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757759.575509:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757759.575521:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757759/real 1787757759] req@ffff8c05a9f78a80 x1874588654989184/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757759.575536:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588654989184/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757759.575549:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757759.575555:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757759.575561:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654989184:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757759.575572:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654989312:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757759.575612:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757759.575636:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757759/real 1787757759] req@ffff8c05a9f78380 x1874588654989312/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757759.575660:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588654989312/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757759.575672:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757759.575675:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757759.575680:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654989312:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757759.575687:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654989440:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757759.575715:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757759.575735:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757759/real 1787757759] req@ffff8c05a9f7b480 x1874588654989440/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757759.575759:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588654989440/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757759.575776:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757759.575783:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757759.575791:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654989440:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757759.702028:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757760.150401:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757760.726031:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757761.174207:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757761.750041:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757762.198339:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757762.774031:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757763.222222:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757763.798024:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757764.246361:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757764.694022:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757764.694035:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757764.694042:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757764.694055:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10821 00000100:00080000:3.0:1787757764.694071:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757764.694081:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757764.694132:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588654989568/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757764.694167:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757764.694172:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757764.694174:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757764.694179:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10821 00000100:00080000:3.0:1787757764.694183:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757764.694188:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757764.694200:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757764.694214:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588654989696/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757764.694226:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757764.694231:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757764.694233:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757764.694237:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10821 00000100:00080000:3.0:1787757764.694240:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757764.694245:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757764.694250:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757764.694262:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588654989824/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757764.694283:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757764.694288:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757764.694292:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757764.694297:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10821 00000100:00080000:3.0:1787757764.694300:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757764.694304:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757764.694311:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757764.694326:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588654989952/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757764.694341:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757764.694346:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1787757764.694350:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654989568:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757764.694350:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757764.694376:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10821 00000100:00080000:3.0:1787757764.694381:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757764.694387:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757764.694393:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757764.694414:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588654990080/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757764.694427:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757764.694432:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757764.694434:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757764.694438:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10821 00000100:00080000:3.0:1787757764.694445:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757764.694450:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757764.694458:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757764.694471:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588654990208/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757764.694482:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757764.694487:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757764.694489:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757764.694494:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10821 00000100:00080000:3.0:1787757764.694499:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757764.694503:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757764.694509:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787757764.694523:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757764.694534:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588654990336/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757764.694547:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757764.694552:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757764.694554:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757764.694561:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10821 00000100:00080000:3.0:1787757764.694566:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757764.694572:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757764.694577:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757764.694595:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588654990464/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787757764.694646:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757764/real 1787757764] req@ffff8c05a9f7ad80 x1874588654989568/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757764.694684:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588654989568/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757764.694715:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757764.694722:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757764.694728:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654989568:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787757764.694740:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654989696:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757764.694779:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757764.694797:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757764/real 1787757764] req@ffff8c05a9f7b800 x1874588654989696/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757764.694866:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588654989696/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757764.694934:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757764.694940:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757764.694945:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654989696:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757764.694954:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654989824:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757764.694988:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757764.695004:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757764/real 1787757764] req@ffff8c05a9f78e00 x1874588654989824/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757764.695020:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588654989824/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757764.695031:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757764.695035:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757764.695039:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654989824:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757764.695082:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654989952:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757764.695106:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757764.695119:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757764/real 1787757764] req@ffff8c05a9f79500 x1874588654989952/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757764.695134:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588654989952/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757764.695145:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757764.695148:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757764.695152:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654989952:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757764.695159:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654990080:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757764.695179:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757764.695191:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757764/real 1787757764] req@ffff8c05a9f7b100 x1874588654990080/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757764.695205:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588654990080/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757764.695215:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757764.695219:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757764.695222:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654990080:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757764.695229:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654990208:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787757764.695248:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757764.695259:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757764/real 1787757764] req@ffff8c05a9f79180 x1874588654990208/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757764.695274:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588654990208/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757764.695284:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757764.695287:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757764.695291:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654990208:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757764.695297:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654990336:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757764.695317:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757764.695328:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757764/real 1787757764] req@ffff8c05a9f79f80 x1874588654990336/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757764.695343:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588654990336/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757764.695353:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757764.695357:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757764.695360:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654990336:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757764.695366:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654990464:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787757764.695385:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757764.695396:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757764/real 1787757764] req@ffff8c05a9f7bb80 x1874588654990464/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757764.695409:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588654990464/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757764.695428:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757764.695432:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757764.695436:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654990464:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757764.822025:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757765.270238:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757765.846014:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757766.294283:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757766.870014:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757767.318194:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757767.893983:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757768.342289:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757768.918010:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757769.366108:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757769.814123:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757769.814139:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787757769.814146:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757769.814163:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10826 00000100:00080000:2.0:1787757769.814181:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757769.814193:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757769.814258:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73800 x1874588654990592/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757769.814310:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757769.814316:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757769.814319:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757769.814324:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10826 00000100:00080000:2.0:1787757769.814329:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757769.814333:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757769.814347:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757769.814367:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588654990720/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757769.814381:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757769.814387:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757769.814390:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757769.814396:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10826 00000100:00080000:2.0:1787757769.814402:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757769.814407:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757769.814413:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757769.814425:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588654990848/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757769.814441:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757769.814448:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757769.814457:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757769.814459:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654990592:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757769.814466:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10826 00000100:00080000:2.0:1787757769.814482:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757769.814491:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757769.814497:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757769.814517:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588654990976/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757769.814530:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757769.814536:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757769.814538:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757769.814547:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10826 00000100:00080000:2.0:1787757769.814564:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757769.814570:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757769.814578:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787757769.814583:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757769.814606:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588654991104/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757769.814623:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757769.814628:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757769.814632:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757769.814639:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10826 00000100:00080000:2.0:1787757769.814644:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757769.814649:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757769.814654:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757769.814671:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757769/real 1787757769] req@ffff8c05b2e73800 x1874588654990592/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757769.814673:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588654991232/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757769.814765:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757769.814770:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787757769.814774:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757769.814798:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10826 00000100:00080000:2.0:1787757769.814808:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757769.814817:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757769.814827:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757769.814846:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73800 x1874588654990592/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757769.814848:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588654991360/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757769.814931:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757769.814939:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757769.814949:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654990592:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757769.814978:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787757769.814979:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654990720:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757769.815004:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757769.815011:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757769.815026:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10826 00000100:00080000:2.0:1787757769.815038:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787757769.815043:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757769.815057:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757769.815074:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757769.815074:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757769/real 1787757769] req@ffff8c05b2e70380 x1874588654990720/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757769.815111:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588654990720/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757769.815131:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588654991488/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757769.815131:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757769.815150:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757769.815165:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654990720:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757769.815228:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654990848:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757769.815260:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757769.815274:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757769/real 1787757769] req@ffff8c05b2e71880 x1874588654990848/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757769.815289:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588654990848/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757769.815298:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757769.815303:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757769.815307:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654990848:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757769.815315:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654990976:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757769.815335:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757769.815346:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757769/real 1787757769] req@ffff8c05b2e71c00 x1874588654990976/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757769.815361:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588654990976/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757769.815371:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757769.815374:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757769.815378:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654990976:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757769.815385:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654991104:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757769.815405:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757769.815416:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757769/real 1787757769] req@ffff8c05b2e73b80 x1874588654991104/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757769.815430:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588654991104/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757769.815439:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757769.815443:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757769.815446:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654991104:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757769.815453:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654991232:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757769.815472:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757769.815482:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757769/real 1787757769] req@ffff8c05b2e71500 x1874588654991232/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757769.815495:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588654991232/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757769.815504:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757769.815508:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757769.815512:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654991232:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757769.815518:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654991360:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757769.815538:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757769.815549:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757769/real 1787757769] req@ffff8c05b2e73480 x1874588654991360/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757769.815572:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588654991360/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757769.815581:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757769.815585:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757769.815588:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654991360:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757769.815595:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654991488:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757769.815614:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757769.815625:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757769/real 1787757769] req@ffff8c05b2e70700 x1874588654991488/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757769.815637:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588654991488/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757769.815647:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757769.815650:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757769.815654:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654991488:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757769.942078:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757770.390298:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757770.966109:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757771.414223:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757771.989996:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757772.438136:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757773.014032:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757773.462198:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757774.038025:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757774.486329:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757774.934140:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757774.934156:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787757774.934164:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757774.934183:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10831 00000100:00080000:2.0:1787757774.934205:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757774.934218:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757774.934285:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588654991616/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757774.934350:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757774.934359:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757774.934364:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757774.934380:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10831 00000100:00080000:2.0:1787757774.934391:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757774.934402:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757774.934422:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757774.934442:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654991616:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787757774.934450:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588654991744/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757774.934484:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757774.934489:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757774.934493:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757774.934500:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10831 00000100:00080000:2.0:1787757774.934512:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757774.934520:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757774.934530:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757774.934564:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588654991872/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757774.934585:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757774.934592:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757774.934596:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757774.934606:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10831 00000100:00080000:2.0:1787757774.934612:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757774.934616:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757774.934655:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757774.934676:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588654992000/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757774.934692:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757774.934698:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757774.934701:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757774.934707:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10831 00000100:00080000:2.0:1787757774.934714:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787757774.934720:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757774.934721:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757774.934731:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757774.934744:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588654992128/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757774.934757:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757774.934762:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757774.934764:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757774.934769:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10831 00000100:00080000:2.0:1787757774.934776:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757774.934780:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757774.934786:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757774.934801:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588654992256/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757774.934811:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757774.934815:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787757774.934817:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757774.934822:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10831 00000100:00080000:2.0:1787757774.934827:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757774.934832:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757774.934838:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757774.934854:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588654992384/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757774.934866:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757774.934872:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757774.934926:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757774.934935:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10831 00000100:00080000:2.0:1787757774.934946:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757774.934956:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757774.934967:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757774.934990:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588654992512/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757774.935135:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757774/real 1787757774] req@ffff8c05b2e70e00 x1874588654991616/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757774.935164:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588654991616/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757774.935180:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757774.935186:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757774.935193:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654991616:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757774.935236:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654991744:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757774.935288:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757774.935314:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757774/real 1787757774] req@ffff8c05b2e70000 x1874588654991744/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757774.935332:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588654991744/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757774.935343:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757774.935348:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757774.935352:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654991744:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757774.935361:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654991872:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757774.935389:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757774.935403:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757774/real 1787757774] req@ffff8c05b2e72300 x1874588654991872/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757774.935417:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588654991872/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757774.935427:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757774.935431:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757774.935435:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654991872:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757774.935444:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654992000:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757774.935464:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757774.935476:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757774/real 1787757774] req@ffff8c05b2e72a00 x1874588654992000/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757774.935490:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588654992000/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757774.935500:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757774.935503:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757774.935507:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654992000:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757774.935514:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654992128:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757774.935533:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757774.935544:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757774/real 1787757774] req@ffff8c05b2e73100 x1874588654992128/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757774.935557:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588654992128/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757774.935567:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757774.935571:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757774.935574:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654992128:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757774.935581:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654992256:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757774.935600:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757774.935611:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757774/real 1787757774] req@ffff8c05b2e72d80 x1874588654992256/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757774.935625:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588654992256/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757774.935669:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757774.935677:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757774.935684:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654992256:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757774.935698:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654992384:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757774.935742:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757774.935763:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757774/real 1787757774] req@ffff8c05b2e72680 x1874588654992384/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757774.935778:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588654992384/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757774.935789:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757774.935793:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757774.935797:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654992384:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757774.935805:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654992512:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757774.935830:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757774.935842:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757774/real 1787757774] req@ffff8c05b2e71180 x1874588654992512/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757774.935856:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588654992512/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757774.935874:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757774.935931:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757774.935936:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654992512:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757775.062016:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757775.510256:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757776.086097:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757776.534305:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757777.110047:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757777.558116:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757778.134064:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757778.582368:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757779.158085:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757779.606143:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757780.054073:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757780.054084:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757780.054089:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757780.054101:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10836 00000100:00080000:3.0:1787757780.054113:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757780.054122:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757780.054172:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588654992640/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757780.054219:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757780.054224:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757780.054227:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757780.054232:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10836 00000100:00080000:3.0:1787757780.054237:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757780.054242:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757780.054254:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757780.054271:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588654992768/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757780.054286:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757780.054292:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757780.054295:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757780.054300:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10836 00000100:00080000:3.0:1787757780.054304:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757780.054308:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757780.054315:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757780.054320:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654992640:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757780.054333:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588654992896/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757780.054348:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757780.054352:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757780.054355:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757780.054360:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10836 00000100:00080000:3.0:1787757780.054364:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757780.054368:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757780.054375:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757780.054390:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588654993024/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757780.054402:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757780.054406:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757780.054409:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757780.054413:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10836 00000800:00000100:0.0:1787757780.054422:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757780.054424:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757780.054435:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757780.054443:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757780.054458:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588654993152/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757780.054469:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757780.054474:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757780.054477:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757780.054484:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10836 00000100:00080000:3.0:1787757780.054491:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757780.054499:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757780.054505:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757780/real 1787757780] req@ffff8c05a9f7bb80 x1874588654992640/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787757780.054514:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757780.054575:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588654993280/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757780.054590:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588654992640/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757780.054609:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757780.054622:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787757780.054625:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757780.054628:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757780.054644:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787757780.054646:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10836 00000100:00080000:3.0:1787757780.054659:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787757780.054664:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654992640:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757780.054672:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757780.054685:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757780.054687:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654992768:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757780.054710:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588654993408/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757780.054725:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757780.054731:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757780.054738:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787757780.054740:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757780.054749:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10836 00000100:00080000:3.0:1787757780.054758:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757780.054764:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757780.054771:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757780.054774:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757780/real 1787757780] req@ffff8c05a9f79f80 x1874588654992768/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757780.054791:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588654993536/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757780.054805:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588654992768/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757780.054830:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757780.054840:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757780.054849:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654992768:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757780.054941:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654992896:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757780.055005:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757780.055031:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757780/real 1787757780] req@ffff8c05a9f79180 x1874588654992896/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757780.055056:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588654992896/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757780.055075:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757780.055079:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757780.055084:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654992896:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757780.055092:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654993024:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757780.055119:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757780.055133:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757780/real 1787757780] req@ffff8c05a9f7b100 x1874588654993024/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757780.055147:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588654993024/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757780.055157:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757780.055161:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757780.055164:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654993024:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757780.055171:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654993152:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757780.055190:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757780.055202:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757780/real 1787757780] req@ffff8c05a9f79500 x1874588654993152/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757780.055216:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588654993152/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757780.055225:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757780.055229:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757780.055233:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654993152:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757780.055240:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654993280:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757780.055259:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757780.055269:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757780/real 1787757780] req@ffff8c05a9f78e00 x1874588654993280/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757780.055283:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588654993280/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757780.055292:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757780.055296:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757780.055299:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654993280:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757780.055305:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654993408:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757780.055322:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757780.055333:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757780/real 1787757780] req@ffff8c05a9f7b800 x1874588654993408/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757780.055346:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588654993408/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757780.055356:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757780.055359:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757780.055362:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654993408:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757780.055368:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654993536:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757780.055386:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757780.055445:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757780/real 1787757780] req@ffff8c05a9f7ad80 x1874588654993536/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757780.055475:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588654993536/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757780.055500:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757780.055506:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757780.055512:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654993536:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757780.182020:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757780.630221:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757781.206028:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757781.654062:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757782.230065:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757782.678257:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757783.254075:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757783.702131:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757784.278029:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757784.726279:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757785.174018:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757785.174031:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757785.174037:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757785.174050:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10841 00000100:00080000:3.0:1787757785.174063:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757785.174072:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757785.174119:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588654993664/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757785.174158:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757785.174162:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757785.174165:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757785.174171:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10841 00000100:00080000:3.0:1787757785.174175:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757785.174179:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757785.174191:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757785.174208:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588654993792/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757785.174224:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757785.174229:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757785.174236:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757785.174238:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654993664:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757785.174246:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10841 00000100:00080000:3.0:1787757785.174256:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757785.174261:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757785.174267:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757785.174284:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588654993920/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757785.174298:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757785.174303:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757785.174306:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757785.174310:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10841 00000100:00080000:3.0:1787757785.174314:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757785.174319:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757785.174325:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787757785.174338:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757785.174344:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588654994048/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757785.174360:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757785.174366:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757785.174368:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757785.174373:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10841 00000100:00080000:3.0:1787757785.174378:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757785.174383:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757785.174391:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757785.174405:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757785/real 1787757785] req@ffff8c05a9f7b480 x1874588654993664/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757785.174413:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588654994176/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757785.174484:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757785.174488:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757785.174491:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757785.174495:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588654993664/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757785.174503:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10841 00000100:00080000:3.0:1787757785.174514:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757785.174525:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787757785.174526:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787757785.174536:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787757785.174539:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757785.174549:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654993664:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757785.174559:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588654994304/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757785.174573:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757785.174578:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757785.174581:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787757785.174588:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654993792:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757785.174594:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10841 00000100:00080000:3.0:1787757785.174605:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757785.174610:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757785.174617:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757785.174642:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588654994432/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757785.174643:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757785.174672:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757785/real 1787757785] req@ffff8c05a9f78380 x1874588654993792/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757785.174674:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757785.174693:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757785.174699:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757785.174716:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10841 00000100:00080000:3.0:1787757785.174729:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757785.174739:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757785.174746:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757785.174749:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588654993792/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757785.174767:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757785.174774:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757785.174779:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588654994560/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757785.174780:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654993792:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757785.174799:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654993920:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757785.174839:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757785.174857:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757785/real 1787757785] req@ffff8c05a9f78a80 x1874588654993920/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757785.174918:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588654993920/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757785.174930:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757785.174934:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757785.174939:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654993920:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757785.174947:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654994048:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757785.174972:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757785.174987:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757785/real 1787757785] req@ffff8c05a9f7a680 x1874588654994048/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757785.175002:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588654994048/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757785.175020:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757785.175024:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757785.175028:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654994048:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757785.175035:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654994176:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757785.175053:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757785.175082:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757785/real 1787757785] req@ffff8c05a9f7aa00 x1874588654994176/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757785.175096:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588654994176/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757785.175107:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757785.175111:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757785.175114:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654994176:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757785.175121:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654994304:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757785.175142:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757785.175154:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757785/real 1787757785] req@ffff8c05a9f7a300 x1874588654994304/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757785.175167:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588654994304/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757785.175177:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757785.175180:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757785.175185:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654994304:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757785.175208:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654994432:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757785.175229:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757785.175241:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757785/real 1787757785] req@ffff8c05a9f79880 x1874588654994432/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757785.175255:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588654994432/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757785.175264:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757785.175268:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757785.175271:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654994432:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757785.175277:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654994560:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757785.175296:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757785.175307:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757785/real 1787757785] req@ffff8c05a9f78700 x1874588654994560/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757785.175321:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588654994560/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757785.175330:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757785.175333:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757785.175337:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654994560:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757785.302034:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757785.750140:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757786.325998:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757786.774263:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757787.350068:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757787.798164:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757788.374024:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757788.822201:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757789.397977:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757789.846131:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757790.294127:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757790.294139:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787757790.294144:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757790.294156:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10846 00000100:00080000:2.0:1787757790.294171:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757790.294180:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757790.294226:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588654994688/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757790.294265:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757790.294270:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757790.294272:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757790.294277:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10846 00000100:00080000:2.0:1787757790.294281:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757790.294285:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757790.294296:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757790.294313:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588654994816/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757790.294326:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757790.294330:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757790.294332:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757790.294337:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10846 00000100:00080000:2.0:1787757790.294341:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757790.294346:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757790.294351:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757790.294406:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588654994944/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757790.294423:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787757790.294426:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654994688:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757790.294434:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757790.294442:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757790.294448:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10846 00000100:00080000:2.0:1787757790.294452:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757790.294456:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757790.294465:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757790.294481:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588654995072/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757790.294495:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757790.294499:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757790.294502:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757790.294506:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10846 00000100:00080000:2.0:1787757790.294511:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757790.294515:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757790.294521:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787757790.294526:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757790.294543:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588654995200/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757790.294557:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757790.294561:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757790.294564:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757790.294569:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10846 00000100:00080000:2.0:1787757790.294574:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757790.294578:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757790.294586:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757790.294588:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757790/real 1787757790] req@ffff8c05b2e70a80 x1874588654994688/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757790.294608:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588654995328/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757790.294623:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588654994688/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757790.294626:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757790.294644:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787757790.294652:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757790.294663:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757790.294665:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10846 00000100:00080000:2.0:1787757790.294675:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757790.294678:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787757790.294687:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757790.294694:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654994688:192.168.201.107@tcp:250:kworker.0 00000008:00100000:2.0:1787757790.294699:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757790.294710:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654994816:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757790.294721:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588654995456/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757790.294736:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757790.294741:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757790.294745:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787757790.294753:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757790.294758:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10846 00000100:00080000:2.0:1787757790.294767:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757790.294773:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757790.294780:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757790.294781:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757790/real 1787757790] req@ffff8c05b2e71180 x1874588654994816/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757790.294801:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588654995584/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757790.294808:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588654994816/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757790.294849:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757790.294855:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757790.294860:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654994816:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757790.294952:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654994944:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757790.294981:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757790.294996:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757790/real 1787757790] req@ffff8c05b2e72680 x1874588654994944/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757790.295021:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588654994944/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757790.295032:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757790.295036:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757790.295040:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654994944:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757790.295047:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654995072:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757790.295066:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757790.295078:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757790/real 1787757790] req@ffff8c05b2e72d80 x1874588654995072/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757790.295092:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588654995072/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757790.295102:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757790.295105:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757790.295108:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654995072:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757790.295115:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654995200:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757790.295134:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757790.295145:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757790/real 1787757790] req@ffff8c05b2e73100 x1874588654995200/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757790.295158:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588654995200/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757790.295176:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757790.295180:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757790.295184:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654995200:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757790.295190:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654995328:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757790.295208:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757790.295219:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757790/real 1787757790] req@ffff8c05b2e72a00 x1874588654995328/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757790.295232:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588654995328/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757790.295242:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757790.295245:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757790.295249:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654995328:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757790.295254:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654995456:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757790.295272:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757790.295282:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757790/real 1787757790] req@ffff8c05b2e72300 x1874588654995456/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757790.295295:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588654995456/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757790.295304:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757790.295307:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757790.295312:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654995456:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757790.295318:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654995584:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757790.295337:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757790.295347:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757790/real 1787757790] req@ffff8c05b2e70000 x1874588654995584/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757790.295391:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588654995584/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757790.295402:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757790.295405:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757790.295409:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654995584:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757790.422004:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757790.870267:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757791.446009:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757791.894251:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757792.469999:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757792.918246:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757793.494024:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757793.942236:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757794.518008:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757794.966217:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757795.414034:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757795.414046:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757795.414052:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757795.414064:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10851 00000100:00080000:3.0:1787757795.414077:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757795.414086:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757795.414133:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588654995712/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757795.414173:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757795.414180:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757795.414184:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757795.414192:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10851 00000100:00080000:3.0:1787757795.414199:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757795.414207:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757795.414222:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757795.414248:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588654995840/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757795.414255:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654995712:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757795.414294:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757795.414312:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757795.414316:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757795.414323:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10851 00000100:00080000:3.0:1787757795.414330:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757795.414336:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757795.414346:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757795.414366:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588654995968/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757795.414369:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757795.414385:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757795.414391:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757795.414395:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757795.414402:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10851 00000100:00080000:3.0:1787757795.414408:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757795.414415:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757795.414423:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757795.414430:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757795/real 1787757795] req@ffff8c05a9f78700 x1874588654995712/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757795.414441:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588654996096/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757795.414458:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787757795.414461:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588654995712/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757795.414465:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757795.414469:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757795.414476:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10851 00000100:00080000:3.0:1787757795.414481:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757795.414482:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757795.414488:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787757795.414489:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757795.414497:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654995712:192.168.201.107@tcp:250:kworker.0 00000008:00100000:3.0:1787757795.414497:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757795.414513:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588654996224/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757795.414530:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787757795.414532:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654995840:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757795.414537:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757795.414540:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757795.414547:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10851 00000100:00080000:3.0:1787757795.414553:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757795.414560:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757795.414569:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757795.414588:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588654996352/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757795.414590:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757795.414606:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757795.414613:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787757795.414615:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757795/real 1787757795] req@ffff8c05a9f79880 x1874588654995840/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757795.414617:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757795.414624:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10851 00000100:00080000:3.0:1787757795.414630:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757795.414637:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757795.414639:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588654995840/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787757795.414647:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787757795.414656:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757795.414663:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757795.414665:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588654996480/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787757795.414671:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654995840:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757795.414684:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654995968:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757795.414684:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757795.414690:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757795.414694:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757795.414700:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10851 00000100:00080000:3.0:1787757795.414705:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757795.414712:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757795.414720:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757795.414738:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588654996608/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787757795.414739:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757795.414759:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757795/real 1787757795] req@ffff8c05a9f7a300 x1874588654995968/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757795.414784:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588654995968/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757795.414800:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757795.414807:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757795.414814:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654995968:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757795.414827:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654996096:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757795.414862:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757795.414921:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757795/real 1787757795] req@ffff8c05a9f7aa00 x1874588654996096/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757795.414939:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588654996096/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757795.414951:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757795.414955:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757795.414960:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654996096:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757795.414983:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654996224:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757795.415009:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757795.415023:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757795/real 1787757795] req@ffff8c05a9f7a680 x1874588654996224/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757795.415038:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588654996224/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757795.415048:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757795.415051:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757795.415055:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654996224:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757795.415062:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654996352:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787757795.415080:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757795.415091:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757795/real 1787757795] req@ffff8c05a9f78a80 x1874588654996352/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757795.415105:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588654996352/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757795.415114:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757795.415117:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757795.415122:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654996352:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757795.415129:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654996480:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757795.415147:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757795.415158:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757795/real 1787757795] req@ffff8c05a9f78380 x1874588654996480/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757795.415171:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588654996480/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757795.415181:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757795.415185:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757795.415188:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654996480:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787757795.415194:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654996608:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787757795.415212:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757795.415222:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757795/real 1787757795] req@ffff8c05a9f7b480 x1874588654996608/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757795.415236:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588654996608/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757795.415246:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757795.415249:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757795.415252:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654996608:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757795.541971:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757795.990220:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757796.565977:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757797.014186:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757797.589989:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757798.038127:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757798.613992:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757799.062261:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757799.638057:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757800.086180:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757800.534053:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757800.534070:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787757800.534077:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757800.534092:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10856 00000100:00080000:0.0:1787757800.534109:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757800.534121:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757800.534205:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588654996736/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757800.534255:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757800.534262:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757800.534266:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757800.534276:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10856 00000100:00080000:0.0:1787757800.534285:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757800.534293:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757800.534305:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757800.534319:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654996736:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787757800.534332:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588654996864/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757800.534350:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757800.534355:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757800.534358:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757800.534363:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10856 00000100:00080000:0.0:1787757800.534368:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757800.534372:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757800.534379:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757800.534395:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588654996992/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757800.534409:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757800.534414:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757800.534416:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757800.534420:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10856 00000100:00080000:0.0:1787757800.534426:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757800.534430:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757800.534436:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757800.534455:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588654997120/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757800.534466:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757800.534471:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757800.534474:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757800.534482:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654996864:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757800.534483:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10856 00000100:00080000:0.0:1787757800.534493:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757800.534500:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757800.534507:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757800.534528:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588654997248/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757800.534543:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787757800.534545:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654996992:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757800.534556:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757800.534562:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757800.534571:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10856 00000100:00080000:0.0:1787757800.534578:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757800.534583:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757800.534590:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654997120:192.168.201.107@tcp:8:kworker.0 00000008:00100000:0.0:1787757800.534595:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757800.534626:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588654997376/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757800.534639:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757800.534643:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787757800.534647:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757800.534657:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10856 00000100:00080000:0.0:1787757800.534662:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787757800.534671:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654997248:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757800.534672:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757800.534685:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757800.534710:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588654997504/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757800.534725:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757800.534731:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757800.534735:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757800.534742:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10856 00000100:00080000:0.0:1787757800.534747:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787757800.534753:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654997376:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757800.534760:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757800.534771:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757800.534792:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588654997632/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757800.534812:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654997504:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757800.534869:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654997632:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757800.662008:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757801.110281:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757801.685996:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757802.134146:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757802.709994:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757803.158203:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787757803.606627:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.107:988 00000400:00000100:2.0:1787757803.606698:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.107@tcp at host 192.168.201.107 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787757803.606727:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787757803.606804:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787757803.606824:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787757803.606840:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787757803.606854:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787757803.606869:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:3.0:1787757803.606938:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757800/real 1787757803] req@ffff8c05a9f77100 x1874588654997248/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787757803.606944:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787757803.606959:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:3.0:1787757803.607015:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588654997248/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757803.607041:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757803.607052:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757803.607065:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654997248:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757803.607084:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757800/real 1787757803] req@ffff8c05a9f75180 x1874588654997376/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757803.607118:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588654997376/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757803.607161:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757803.607170:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757803.607181:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654997376:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757803.607197:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757800/real 1787757803] req@ffff8c05a9f74000 x1874588654996864/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757803.607242:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588654996864/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757803.607258:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757803.607267:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757803.607275:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654996864:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757803.608148:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757800/real 1787757803] req@ffff8c05a9f75880 x1874588654996992/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757803.608185:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588654996992/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757803.608205:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757803.608214:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757803.608226:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654996992:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757803.608242:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757800/real 1787757803] req@ffff8c05a9f75500 x1874588654997120/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757803.608269:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588654997120/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757803.608287:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757803.608381:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757803.608392:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654997120:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757803.608407:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757800/real 1787757803] req@ffff8c05a9f74e00 x1874588654996736/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757803.608435:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588654996736/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757803.608451:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757803.608457:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757803.608464:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654996736:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757803.608505:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757800/real 1787757803] req@ffff8c05a9f76d80 x1874588654997632/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757803.608521:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588654997632/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757803.608532:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757803.608536:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757803.608540:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654997632:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757803.608548:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757800/real 1787757803] req@ffff8c05a9f76300 x1874588654997504/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757803.608561:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588654997504/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757803.608570:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757803.608574:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757803.608578:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654997504:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757803.734012:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757804.182081:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757804.758015:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757805.206251:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757805.654002:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757805.654016:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787757805.654023:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757805.654038:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10862 00000100:00080000:0.0:1787757805.654055:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757805.654067:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757805.654151:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588654997760/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757805.654200:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757805.654206:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757805.654210:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757805.654228:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10862 00000100:00080000:0.0:1787757805.654236:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757805.654244:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757805.654261:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757805.654289:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588654997888/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757805.654302:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654997760:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757805.654317:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757805.654327:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757805.654329:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757805.654338:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10862 00000100:00080000:0.0:1787757805.654344:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757805.654351:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757805.654360:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757805.654381:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588654998016/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757805.654394:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757805.654399:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757805.654401:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757805.654407:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10862 00000100:00080000:0.0:1787757805.654414:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787757805.654417:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757805.654426:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757805.654439:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757805.654462:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588654998144/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757805.654481:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757805.654487:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757805.654493:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757805.654504:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10862 00000100:00080000:0.0:1787757805.654511:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787757805.654517:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757805/real 1787757805] req@ffff8c05a9f74380 x1874588654997760/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757805.654520:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757805.654535:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757805.654568:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588654998272/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757805.654581:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588654997760/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757805.654602:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757805.654618:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757805.654623:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757805.654624:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757805.654635:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10862 00000100:00080000:3.0:1787757805.654645:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787757805.654650:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787757805.654659:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654997760:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757805.654669:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757805.654685:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757805.654709:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588654998400/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757805.654720:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654997888:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757805.654751:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757805.654759:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787757805.654763:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757805.654771:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10862 00000100:00080000:0.0:1787757805.654781:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757805.654793:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:3.0:1787757805.654795:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:0.0:1787757805.654811:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757805.654839:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757805/real 1787757805] req@ffff8c05a9f76a00 x1874588654997888/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757805.654849:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588654998528/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757805.654930:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757805.654936:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757805.654939:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757805.654947:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10862 00000100:00080000:0.0:1787757805.654955:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757805.654970:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757805.654984:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757805.654993:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588654997888/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757805.655016:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588654998656/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757805.655022:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757805.655039:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757805.655057:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654997888:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757805.655438:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654998016:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757805.655497:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757805.655524:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757805/real 1787757805] req@ffff8c05a9f77b80 x1874588654998016/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757805.655551:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588654998016/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757805.655568:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757805.655575:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757805.655582:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654998016:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757805.655608:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654998144:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757805.655665:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757805.655692:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757805/real 1787757805] req@ffff8c05a9f76680 x1874588654998144/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757805.655717:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588654998144/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757805.655746:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757805.655753:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757805.655760:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654998144:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757805.655773:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654998272:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757805.655815:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757805.655838:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757805/real 1787757805] req@ffff8c05a9f77800 x1874588654998272/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757805.655861:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588654998272/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757805.655914:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757805.655922:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757805.655929:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654998272:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757805.655969:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654998400:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757805.656014:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757805.656037:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757805/real 1787757805] req@ffff8c05a9f75c00 x1874588654998400/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757805.656063:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588654998400/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757805.656080:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757805.656227:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757805.656237:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654998400:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757805.656246:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654998528:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757805.656281:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757805.656297:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757805/real 1787757805] req@ffff8c05a9f74a80 x1874588654998528/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757805.656313:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588654998528/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757805.656323:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757805.656327:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757805.656331:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654998528:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757805.656337:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654998656:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757805.656357:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757805.656368:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757805/real 1787757805] req@ffff8c05a9f74380 x1874588654998656/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757805.656381:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588654998656/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757805.656391:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757805.656394:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757805.656398:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654998656:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757805.782012:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757806.230187:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757806.805977:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757807.254217:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757807.829999:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757808.278141:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757808.854014:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757809.302210:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757809.878037:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757810.326154:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757810.774285:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757810.774303:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787757810.774312:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757810.774334:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10867 00000100:00080000:0.0:1787757810.774355:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757810.774370:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757810.774448:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588654998784/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757810.774487:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757810.774494:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757810.774497:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757810.774504:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10867 00000100:00080000:0.0:1787757810.774516:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757810.774523:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757810.774545:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757810.774573:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588654998912/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757810.774582:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654998784:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757810.774605:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757810.774611:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757810.774616:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757810.774623:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10867 00000100:00080000:0.0:1787757810.774628:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757810.774634:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757810.774641:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757810.774661:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588654999040/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757810.774676:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757810.774680:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757810.774682:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757810.774687:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10867 00000100:00080000:0.0:1787757810.774695:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757810.774699:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757810.774705:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787757810.774723:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757810.774726:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588654999168/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757810.774745:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757810.774750:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757810.774754:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757810.774760:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10867 00000100:00080000:0.0:1787757810.774782:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757810.774788:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757810.774795:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757810.774828:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757810/real 1787757810] req@ffff8c05a9f76300 x1874588654998784/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757810.774829:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588654999296/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757810.774940:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757810.774951:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757810.774956:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787757810.774965:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588654998784/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757810.774976:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10867 00000100:00080000:0.0:1787757810.775018:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757810.775022:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757810.775035:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:1.0:1787757810.775036:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757810.775047:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654998784:192.168.201.107@tcp:250:kworker.0 00000008:00100000:0.0:1787757810.775057:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757810.775087:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588654999424/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757810.775097:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654998912:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757810.775116:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757810.775123:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787757810.775126:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757810.775135:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10867 00000100:00080000:0.0:1787757810.775147:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787757810.775155:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757810.775157:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757810.775172:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757810.775188:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757810/real 1787757810] req@ffff8c05a9f76d80 x1874588654998912/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757810.775197:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588654999552/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757810.775238:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787757810.775241:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588654998912/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757810.775256:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757810.775265:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757810.775285:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10867 00000100:00080000:1.0:1787757810.775286:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757810.775296:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757810.775300:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787757810.775309:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757810.775315:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654998912:192.168.201.107@tcp:38:kworker.0 00000008:00100000:0.0:1787757810.775321:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757810.775331:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654999040:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757810.775345:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588654999680/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787757810.775378:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757810.775411:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757810/real 1787757810] req@ffff8c05a9f74e00 x1874588654999040/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757810.775438:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588654999040/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757810.775455:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757810.775467:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757810.775481:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654999040:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757810.775510:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654999168:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757810.775570:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757810.775621:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757810/real 1787757810] req@ffff8c05a9f75500 x1874588654999168/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757810.775660:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588654999168/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757810.775685:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757810.775693:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757810.775704:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654999168:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757810.775724:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654999296:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757810.775776:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757810.775804:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757810/real 1787757810] req@ffff8c05a9f75880 x1874588654999296/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757810.775821:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588654999296/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757810.775833:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757810.775837:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757810.775841:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654999296:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757810.775868:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654999424:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757810.775942:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757810.775977:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757810/real 1787757810] req@ffff8c05a9f74000 x1874588654999424/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757810.775994:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588654999424/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757810.776033:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757810.776038:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757810.776043:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654999424:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757810.776052:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654999552:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757810.776081:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757810.776095:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757810/real 1787757810] req@ffff8c05a9f75180 x1874588654999552/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757810.776110:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588654999552/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757810.776121:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757810.776125:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757810.776129:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654999552:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757810.776136:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654999680:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757810.776154:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757810.776165:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757810/real 1787757810] req@ffff8c05a9f77100 x1874588654999680/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757810.776179:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588654999680/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757810.776189:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757810.776193:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757810.776196:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588654999680:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757810.901984:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757811.350261:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757811.926091:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757812.374144:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757812.950024:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757813.398214:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757813.974031:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757814.422186:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757814.998036:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757815.446195:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757815.894091:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757815.894104:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787757815.894109:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757815.894121:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10872 00000100:00080000:2.0:1787757815.894134:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757815.894143:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757815.894193:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588654999808/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757815.894234:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757815.894239:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757815.894241:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757815.894247:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10872 00000100:00080000:2.0:1787757815.894251:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757815.894269:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757815.894280:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757815.894301:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588654999936/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757815.894326:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757815.894330:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757815.894332:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757815.894341:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10872 00000100:00080000:2.0:1787757815.894349:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787757815.894354:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654999808:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757815.894355:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757815.894362:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757815.894387:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588655000064/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757815.894400:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757815.894405:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757815.894409:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757815.894414:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10872 00000100:00080000:2.0:1787757815.894418:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757815.894423:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757815.894429:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757815.894444:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588655000192/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757815.894456:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757815.894462:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757815.894465:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757815.894470:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10872 00000100:00080000:2.0:1787757815.894476:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757815.894480:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757815.894486:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757815.894498:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588655000320/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757815.894508:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757815.894513:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757815.894520:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757815.894527:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10872 00000100:00080000:2.0:1787757815.894531:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757815.894535:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757815.894541:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787757815.894544:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757815.894556:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588655000448/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757815.894571:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757815.894578:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787757815.894581:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757815.894587:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10872 00000100:00080000:2.0:1787757815.894593:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757815.894598:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757815.894605:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757815.894622:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588655000576/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757815.894645:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757815/real 1787757815] req@ffff8c05b2e70e00 x1874588654999808/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757815.894655:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757815.894684:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757815.894696:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757815.894722:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10872 00000100:00080000:2.0:1787757815.894728:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757815.894736:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757815.894742:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588654999808/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787757815.894745:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757815.894770:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588655000704/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757815.894804:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757815.894819:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757815.894830:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588654999808:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787757815.894961:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654999936:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757815.895006:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757815.895027:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757815/real 1787757815] req@ffff8c05b2e70700 x1874588654999936/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757815.895046:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588654999936/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757815.895058:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757815.895063:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757815.895068:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588654999936:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757815.895077:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655000064:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757815.895099:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757815.895112:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757815/real 1787757815] req@ffff8c05b2e73480 x1874588655000064/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757815.895127:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588655000064/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757815.895137:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757815.895141:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757815.895144:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655000064:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757815.895152:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655000192:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757815.895174:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757815.895186:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757815/real 1787757815] req@ffff8c05b2e71500 x1874588655000192/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757815.895201:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588655000192/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757815.895212:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757815.895215:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757815.895219:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655000192:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757815.895226:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655000320:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757815.895246:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757815.895258:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757815/real 1787757815] req@ffff8c05b2e73b80 x1874588655000320/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757815.895272:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588655000320/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757815.895282:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757815.895286:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757815.895290:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655000320:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757815.895296:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655000448:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757815.895315:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757815.895330:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757815/real 1787757815] req@ffff8c05b2e71c00 x1874588655000448/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757815.895354:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588655000448/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757815.895371:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757815.895378:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757815.895384:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655000448:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757815.895397:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655000576:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757815.895437:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757815.895455:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757815/real 1787757815] req@ffff8c05b2e71880 x1874588655000576/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757815.895473:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588655000576/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757815.895490:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757815.895497:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757815.895504:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655000576:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757815.895517:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655000704:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757815.895557:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757815.895579:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757815/real 1787757815] req@ffff8c05b2e70380 x1874588655000704/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757815.895595:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588655000704/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757815.895606:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757815.895610:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757815.895615:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655000704:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757816.022003:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757816.470116:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757817.045986:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757817.494212:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757818.070027:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757818.518101:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757819.093984:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757819.542318:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757820.118026:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757820.566152:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757821.014027:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757821.014045:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787757821.014052:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757821.014068:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10877 00000100:00080000:2.0:1787757821.014084:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757821.014097:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757821.014154:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73800 x1874588655000832/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757821.014193:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757821.014197:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757821.014200:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757821.014206:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10877 00000100:00080000:2.0:1787757821.014212:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757821.014217:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757821.014229:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757821.014253:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588655000960/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757821.014270:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757821.014275:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757821.014280:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787757821.014294:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655000832:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757821.014294:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10877 00000100:00080000:2.0:1787757821.014309:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757821.014323:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757821.014329:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757821.014350:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588655001088/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757821.014363:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757821.014368:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757821.014372:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757821.014377:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10877 00000100:00080000:2.0:1787757821.014382:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757821.014386:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757821.014395:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757821.014406:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588655001216/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757821.014419:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757821.014423:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757821.014427:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757821.014437:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10877 00000800:00000100:1.0:1787757821.014443:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757821.014448:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757821.014457:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757821.014464:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757821.014478:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588655001344/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757821.014490:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757821.014494:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757821.014496:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757821.014501:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10877 00000100:00080000:2.0:1787757821.014505:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757821.014511:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757821.014518:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757821.014559:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588655001472/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757821.014560:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757821/real 1787757821] req@ffff8c05b2e73800 x1874588655000832/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757821.014650:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757821.014664:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787757821.014668:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787757821.014681:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73800 x1874588655000832/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757821.014683:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10877 00000100:00080000:2.0:1787757821.014705:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757821.014720:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:1.0:1787757821.014721:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757821.014738:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000008:00100000:2.0:1787757821.014740:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757821.014750:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655000832:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787757821.014773:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588655001600/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757821.014788:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757821.014793:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757821.014797:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787757821.014808:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655000960:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757821.014808:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10877 00000100:00080000:2.0:1787757821.014858:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757821.014869:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757821.014943:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787757821.014956:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757821.014986:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588655001728/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757821.014988:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757821/real 1787757821] req@ffff8c05b2e70380 x1874588655000960/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757821.015056:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588655000960/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757821.015080:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757821.015088:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757821.015098:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655000960:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757821.015120:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655001088:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757821.015164:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757821.015183:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757821/real 1787757821] req@ffff8c05b2e71880 x1874588655001088/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757821.015200:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588655001088/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757821.015211:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757821.015215:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757821.015220:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655001088:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757821.015228:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655001216:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757821.015253:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757821.015264:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757821/real 1787757821] req@ffff8c05b2e71c00 x1874588655001216/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757821.015279:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588655001216/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757821.015289:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757821.015293:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757821.015296:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655001216:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757821.015304:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655001344:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757821.015326:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757821.015338:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757821/real 1787757821] req@ffff8c05b2e73b80 x1874588655001344/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757821.015353:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588655001344/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757821.015363:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757821.015367:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757821.015370:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655001344:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757821.015377:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655001472:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757821.015397:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757821.015408:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757821/real 1787757821] req@ffff8c05b2e71500 x1874588655001472/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757821.015423:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588655001472/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757821.015433:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757821.015436:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757821.015440:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655001472:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757821.015446:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655001600:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757821.015470:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757821.015489:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757821/real 1787757821] req@ffff8c05b2e73480 x1874588655001600/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757821.015511:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588655001600/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757821.015543:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757821.015550:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757821.015557:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655001600:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757821.015595:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655001728:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757821.015625:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757821.015640:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757821/real 1787757821] req@ffff8c05b2e70700 x1874588655001728/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757821.015655:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588655001728/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757821.015666:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757821.015671:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757821.015674:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655001728:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757821.142001:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757821.590251:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757822.165970:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757822.614110:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757823.190084:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757823.638197:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757824.214021:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757824.662097:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757825.237992:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757825.686218:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757826.134029:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757826.134042:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787757826.134048:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757826.134060:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10882 00000100:00080000:2.0:1787757826.134073:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757826.134083:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757826.134133:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588655001856/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757826.134175:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757826.134180:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757826.134182:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757826.134187:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10882 00000100:00080000:2.0:1787757826.134191:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757826.134197:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757826.134208:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757826.134224:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588655001984/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757826.134237:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757826.134241:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757826.134243:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757826.134247:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10882 00000100:00080000:2.0:1787757826.134265:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757826.134279:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757826.134286:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757826.134300:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588655002112/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757826.134316:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757826.134325:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787757826.134330:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655001856:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757826.134330:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757826.134346:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10882 00000100:00080000:2.0:1787757826.134360:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757826.134367:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757826.134373:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757826.134393:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588655002240/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757826.134406:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757826.134413:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757826.134415:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757826.134421:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10882 00000100:00080000:2.0:1787757826.134428:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757826.134433:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757826.134439:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757826.134453:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588655002368/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757826.134464:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757826.134468:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757826.134470:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757826.134474:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10882 00000100:00080000:2.0:1787757826.134479:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787757826.134494:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757826.134494:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757826.134513:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757826.134527:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588655002496/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757826.134540:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757826.134545:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787757826.134548:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757826.134553:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10882 00000100:00080000:2.0:1787757826.134557:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757826.134562:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757826.134571:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757826.134585:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588655002624/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757826.134597:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787757826.134607:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757826/real 1787757826] req@ffff8c05b2e70e00 x1874588655001856/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757826.134608:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757826.134618:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757826.134643:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10882 00000100:00080000:2.0:1787757826.134660:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757826.134676:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757826.134684:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757826.134693:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588655001856/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757826.134702:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588655002752/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757826.134777:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757826.134797:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757826.134809:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655001856:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787757826.134829:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655001984:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757826.134950:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757826.134980:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757826/real 1787757826] req@ffff8c05b2e70000 x1874588655001984/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757826.134999:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588655001984/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757826.135011:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757826.135015:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757826.135020:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655001984:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757826.135059:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655002112:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757826.135090:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757826.135104:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757826/real 1787757826] req@ffff8c05b2e72300 x1874588655002112/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757826.135119:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588655002112/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757826.135129:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757826.135134:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757826.135138:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655002112:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757826.135146:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655002240:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757826.135168:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757826.135180:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757826/real 1787757826] req@ffff8c05b2e72a00 x1874588655002240/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757826.135194:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588655002240/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757826.135204:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757826.135208:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757826.135212:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655002240:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757826.135219:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655002368:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757826.135239:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757826.135250:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757826/real 1787757826] req@ffff8c05b2e73100 x1874588655002368/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757826.135264:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588655002368/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757826.135274:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757826.135278:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757826.135281:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655002368:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757826.135288:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655002496:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757826.135307:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757826.135318:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757826/real 1787757826] req@ffff8c05b2e72d80 x1874588655002496/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757826.135340:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588655002496/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757826.135351:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757826.135354:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757826.135357:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655002496:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757826.135364:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655002624:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757826.135383:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757826.135394:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757826/real 1787757826] req@ffff8c05b2e72680 x1874588655002624/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757826.135407:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588655002624/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757826.135417:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757826.135420:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757826.135424:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655002624:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757826.135430:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655002752:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757826.135454:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757826.135472:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757826/real 1787757826] req@ffff8c05b2e71180 x1874588655002752/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757826.135494:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588655002752/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757826.135523:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757826.135530:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757826.135537:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655002752:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757826.261996:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757826.710128:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757827.286002:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757827.734232:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757828.310000:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757828.758087:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757829.333999:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757829.782268:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757830.358000:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757830.806153:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757831.254008:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757831.254021:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787757831.254027:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757831.254039:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10887 00000100:00080000:2.0:1787757831.254053:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757831.254062:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757831.254108:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588655002880/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757831.254158:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757831.254164:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757831.254166:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757831.254171:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10887 00000100:00080000:2.0:1787757831.254177:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757831.254182:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757831.254193:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757831.254212:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588655003008/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757831.254225:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757831.254229:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757831.254231:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757831.254236:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10887 00000100:00080000:2.0:1787757831.254242:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757831.254247:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757831.254253:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757831.254266:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588655003136/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757831.254280:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655002880:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757831.254286:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757831.254297:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757831.254300:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757831.254306:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10887 00000100:00080000:2.0:1787757831.254310:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757831.254316:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757831.254323:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757831.254339:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588655003264/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757831.254361:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757831.254366:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757831.254368:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757831.254372:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10887 00000100:00080000:2.0:1787757831.254377:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787757831.254381:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757831.254390:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757831.254398:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757831.254421:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588655003392/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757831.254441:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787757831.254446:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757831/real 1787757831] req@ffff8c05b2e70a80 x1874588655002880/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757831.254453:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757831.254462:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757831.254480:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10887 00000100:00100000:1.0:1787757831.254488:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588655002880/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757831.254493:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757831.254513:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:1.0:1787757831.254514:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757831.254528:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000008:00100000:2.0:1787757831.254529:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757831.254538:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655002880:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787757831.254551:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655003008:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757831.254560:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588655003520/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757831.254576:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757831.254582:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1787757831.254586:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757831.254589:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757831.254601:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10887 00000100:00080000:2.0:1787757831.254610:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787757831.254616:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757831/real 1787757831] req@ffff8c05b2e71180 x1874588655003008/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757831.254623:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757831.254632:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757831.254694:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588655003008/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757831.254695:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588655003648/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757831.254738:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787757831.254746:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757831.254748:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787757831.254759:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787757831.254762:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655003008:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757831.254765:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757831.254779:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10887 00000100:00080000:2.0:1787757831.254785:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757831.254791:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757831.254798:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757831.254816:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655003136:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757831.254817:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588655003776/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787757831.254950:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757831.254981:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757831/real 1787757831] req@ffff8c05b2e72680 x1874588655003136/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757831.255007:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588655003136/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757831.255020:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757831.255025:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757831.255030:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655003136:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757831.255038:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655003264:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757831.255070:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757831.255084:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757831/real 1787757831] req@ffff8c05b2e72d80 x1874588655003264/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757831.255099:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588655003264/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757831.255109:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757831.255113:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757831.255117:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655003264:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757831.255123:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655003392:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757831.255142:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757831.255154:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757831/real 1787757831] req@ffff8c05b2e73100 x1874588655003392/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757831.255168:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588655003392/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757831.255178:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757831.255181:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757831.255185:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655003392:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757831.255191:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655003520:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757831.255208:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757831.255219:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757831/real 1787757831] req@ffff8c05b2e72a00 x1874588655003520/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757831.255233:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588655003520/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757831.255242:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757831.255245:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757831.255249:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655003520:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757831.255254:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655003648:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757831.255272:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757831.255282:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757831/real 1787757831] req@ffff8c05b2e72300 x1874588655003648/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757831.255295:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588655003648/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757831.255305:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757831.255309:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757831.255313:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655003648:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757831.255334:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655003776:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757831.255353:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757831.255364:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757831/real 1787757831] req@ffff8c05b2e70000 x1874588655003776/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757831.255378:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588655003776/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757831.255388:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757831.255392:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757831.255396:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655003776:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757831.381984:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757831.830215:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757832.405944:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757832.854021:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757833.429985:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757833.878219:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757834.454022:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757834.902144:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757835.477995:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757835.926218:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757836.374076:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757836.374093:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787757836.374101:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757836.374116:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10892 00000100:00080000:0.0:1787757836.374132:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757836.374144:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757836.374201:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588655003904/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757836.374249:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757836.374253:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757836.374256:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757836.374262:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10892 00000100:00080000:0.0:1787757836.374268:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757836.374275:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757836.374287:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757836.374309:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588655004032/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757836.374323:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757836.374327:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757836.374330:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757836.374335:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10892 00000100:00080000:0.0:1787757836.374339:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757836.374350:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757836.374350:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655003904:192.168.201.107@tcp:250:kworker.0 00000008:00100000:0.0:1787757836.374366:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757836.374379:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588655004160/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757836.374394:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757836.374399:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757836.374401:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757836.374407:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10892 00000100:00080000:0.0:1787757836.374412:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757836.374416:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757836.374423:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757836.374436:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655004288/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757836.374448:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787757836.374457:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757836.374460:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757836.374466:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757836.374473:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10892 00000100:00080000:0.0:1787757836.374481:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757836.374488:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757836.374493:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757836.374508:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588655004416/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757836.374523:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787757836.374524:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757836/real 1787757836] req@ffff8c05a9f77100 x1874588655003904/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757836.374532:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757836.374539:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757836.374577:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10892 00000100:00080000:0.0:1787757836.374594:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787757836.374608:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588655003904/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757836.374610:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757836.374629:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787757836.374651:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757836.374659:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757836.374671:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655003904:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787757836.374673:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588655004544/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757836.374686:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655004032:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757836.374713:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757836.374719:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787757836.374723:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757836.374732:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10892 00000800:00000100:1.0:1787757836.374741:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757836.374744:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757836.374756:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757836.374766:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757836/real 1787757836] req@ffff8c05a9f75180 x1874588655004032/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787757836.374767:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757836.374800:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588655004672/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757836.374802:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588655004032/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757836.374864:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757836.374867:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757836.374919:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757836.374926:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655004032:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757836.374957:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757836.374961:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787757836.374964:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655004160:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757836.374972:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10892 00000100:00080000:0.0:1787757836.374981:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757836.374987:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757836.374995:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787757836.375008:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757836.375020:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655004800/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757836.375035:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757836/real 1787757836] req@ffff8c05a9f74000 x1874588655004160/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757836.375062:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588655004160/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757836.375076:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757836.375081:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757836.375086:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655004160:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757836.375096:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655004288:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757836.375118:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757836.375130:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757836/real 1787757836] req@ffff8c05a9f75880 x1874588655004288/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757836.375145:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655004288/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757836.375155:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757836.375158:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757836.375162:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655004288:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757836.375168:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655004416:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757836.375187:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757836.375199:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757836/real 1787757836] req@ffff8c05a9f75500 x1874588655004416/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757836.375212:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588655004416/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757836.375226:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757836.375232:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757836.375238:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655004416:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757836.375250:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655004544:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757836.375287:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757836.375311:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757836/real 1787757836] req@ffff8c05a9f74e00 x1874588655004544/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757836.375338:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588655004544/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757836.375352:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757836.375356:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757836.375360:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655004544:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757836.375367:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655004672:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757836.375396:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757836.375410:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757836/real 1787757836] req@ffff8c05a9f76d80 x1874588655004672/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757836.375425:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588655004672/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757836.375434:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757836.375438:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757836.375441:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655004672:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757836.375464:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655004800:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757836.375484:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757836.375496:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757836/real 1787757836] req@ffff8c05a9f76300 x1874588655004800/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757836.375510:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655004800/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757836.375528:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757836.375532:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757836.375536:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655004800:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757836.501989:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757836.950151:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757837.525951:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757837.974118:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757838.550086:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757838.998175:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757839.574034:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757840.022165:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757840.598033:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757841.046185:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757841.493995:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757841.494009:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787757841.494014:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757841.494039:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10897 00000100:00080000:2.0:1787757841.494052:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757841.494068:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757841.494112:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588655004928/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757841.494162:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757841.494170:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757841.494173:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757841.494182:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10897 00000100:00080000:2.0:1787757841.494189:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757841.494198:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757841.494213:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757841.494239:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588655005056/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757841.494254:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757841.494259:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757841.494263:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757841.494268:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10897 00000100:00080000:2.0:1787757841.494274:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757841.494280:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757841.494287:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757841.494303:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655004928:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787757841.494306:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588655005184/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757841.494331:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757841.494336:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757841.494339:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757841.494345:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10897 00000100:00080000:2.0:1787757841.494351:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757841.494356:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757841.494363:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757841.494378:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588655005312/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757841.494391:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757841.494395:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757841.494399:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787757841.494408:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757841.494415:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10897 00000100:00080000:2.0:1787757841.494424:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757841.494430:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757841.494438:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757841.494453:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588655005440/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757841.494505:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757841.494512:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787757841.494515:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757841/real 1787757841] req@ffff8c05b2e70e00 x1874588655004928/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757841.494524:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757841.494546:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10897 00000100:00080000:2.0:1787757841.494553:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787757841.494558:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588655004928/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757841.494565:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757841.494582:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787757841.494589:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757841.494597:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757841.494606:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588655005568/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757841.494608:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655004928:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787757841.494631:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655005056:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757841.494656:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757841.494662:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787757841.494665:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757841.494673:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10897 00000100:00080000:2.0:1787757841.494680:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787757841.494681:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757841.494691:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757841.494701:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757841.494709:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757841/real 1787757841] req@ffff8c05b2e70700 x1874588655005056/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787757841.494722:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588655005696/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757841.494739:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588655005056/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757841.494743:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757841.494764:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757841.494774:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757841.494775:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757841.494791:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787757841.494797:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10897 00000100:00100000:1.0:1787757841.494803:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655005056:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787757841.494814:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757841.494828:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757841.494839:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757841.494845:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655005184:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757841.494912:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588655005824/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787757841.494938:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757841.494963:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757841/real 1787757841] req@ffff8c05b2e73480 x1874588655005184/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757841.494980:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588655005184/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757841.494991:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757841.494995:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757841.494999:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655005184:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757841.495006:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655005312:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757841.495029:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757841.495042:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757841/real 1787757841] req@ffff8c05b2e71500 x1874588655005312/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757841.495056:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588655005312/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757841.495066:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757841.495070:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757841.495074:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655005312:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757841.495080:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655005440:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757841.495098:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757841.495110:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757841/real 1787757841] req@ffff8c05b2e73b80 x1874588655005440/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757841.495124:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588655005440/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757841.495134:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757841.495137:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757841.495141:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655005440:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757841.495147:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655005568:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757841.495165:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757841.495176:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757841/real 1787757841] req@ffff8c05b2e71c00 x1874588655005568/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757841.495190:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588655005568/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757841.495200:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757841.495203:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757841.495207:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655005568:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757841.495212:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655005696:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757841.495230:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757841.495240:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757841/real 1787757841] req@ffff8c05b2e71880 x1874588655005696/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757841.495253:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588655005696/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757841.495263:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757841.495267:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757841.495271:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655005696:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757841.495291:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655005824:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757841.495310:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757841.495321:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757841/real 1787757841] req@ffff8c05b2e70380 x1874588655005824/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757841.495335:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588655005824/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757841.495345:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757841.495348:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757841.495352:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655005824:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757841.621989:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757842.070250:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757842.646045:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757843.094115:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757843.670041:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757844.118205:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757844.694011:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757845.142191:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757845.718047:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757846.166240:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757846.614029:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757846.614045:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787757846.614052:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757846.614069:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10903 00000100:00080000:2.0:1787757846.614086:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757846.614100:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757846.614167:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73800 x1874588655005952/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757846.614221:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757846.614227:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757846.614230:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757846.614235:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10903 00000100:00080000:2.0:1787757846.614240:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757846.614246:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757846.614262:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757846.614285:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588655006080/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757846.614304:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757846.614311:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757846.614320:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757846.614321:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655005952:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757846.614333:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10903 00000100:00080000:2.0:1787757846.614340:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757846.614345:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757846.614353:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757846.614403:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588655006208/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757846.614419:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757846.614426:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757846.614430:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757846.614436:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10903 00000100:00080000:2.0:1787757846.614457:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757846.614464:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757846.614472:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787757846.614484:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787757846.614494:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588655006336/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757846.614510:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757846.614515:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757846.614519:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757846.614527:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10903 00000100:00080000:2.0:1787757846.614533:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757846.614539:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757846.614545:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757846.614554:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588655006464/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757846.614567:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757846.614571:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757846.614574:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757846.614580:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10903 00000100:00080000:2.0:1787757846.614585:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757846.614589:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757846.614595:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757846.614610:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588655006592/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757846.614622:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757846.614627:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787757846.614630:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757846.614636:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10903 00000100:00080000:2.0:1787757846.614640:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757846.614646:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757846.614652:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757846.614663:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588655006720/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757846.614672:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757846.614676:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757846.614678:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757846.614682:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10903 00000100:00080000:2.0:1787757846.614686:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757846.614690:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757846.614696:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757846.614705:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588655006848/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757846.615009:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757846/real 1787757846] req@ffff8c05b2e73800 x1874588655005952/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757846.615048:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73800 x1874588655005952/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757846.615066:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757846.615073:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757846.615183:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655005952:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757846.615221:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655006080:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757846.615264:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757846.615283:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757846/real 1787757846] req@ffff8c05b2e70380 x1874588655006080/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757846.615310:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588655006080/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757846.615322:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757846.615326:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757846.615331:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655006080:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757846.615338:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655006208:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757846.615361:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757846.615399:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757846/real 1787757846] req@ffff8c05b2e71880 x1874588655006208/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757846.615415:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588655006208/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757846.615426:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757846.615430:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757846.615434:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655006208:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757846.615442:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655006336:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757846.615466:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757846.615478:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757846/real 1787757846] req@ffff8c05b2e71c00 x1874588655006336/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757846.615492:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588655006336/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757846.615511:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757846.615515:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757846.615519:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655006336:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757846.615526:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655006464:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757846.615545:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757846.615557:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757846/real 1787757846] req@ffff8c05b2e73b80 x1874588655006464/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757846.615570:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588655006464/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757846.615581:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757846.615584:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757846.615588:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655006464:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757846.615594:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655006592:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757846.615613:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757846.615623:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757846/real 1787757846] req@ffff8c05b2e71500 x1874588655006592/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757846.615638:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588655006592/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757846.615647:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757846.615650:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757846.615655:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655006592:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757846.615662:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655006720:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757846.615681:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757846.615692:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757846/real 1787757846] req@ffff8c05b2e73480 x1874588655006720/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757846.615705:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588655006720/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757846.615715:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757846.615719:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757846.615722:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655006720:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757846.615728:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655006848:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757846.615746:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757846.615757:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757846/real 1787757846] req@ffff8c05b2e70700 x1874588655006848/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757846.615769:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588655006848/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757846.615779:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757846.615783:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757846.615786:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655006848:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757846.742060:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757847.190198:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757847.766095:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757848.214263:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757848.790073:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757849.238232:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757849.814072:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757850.262258:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757850.838061:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757851.286257:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757851.734104:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757851.734122:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757851.734129:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757851.734147:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10908 00000100:00080000:3.0:1787757851.734164:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757851.734174:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757851.734223:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655006976/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757851.734266:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757851.734271:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757851.734274:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757851.734312:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10908 00000100:00080000:3.0:1787757851.734319:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757851.734326:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757851.734340:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757851.734368:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655006976:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757851.734368:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655007104/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757851.734399:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757851.734403:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757851.734407:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757851.734412:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10908 00000100:00080000:3.0:1787757851.734417:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757851.734422:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757851.734431:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757851.734447:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655007232/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757851.734462:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757851.734471:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1787757851.734472:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757851.734476:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757851.734486:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10908 00000100:00080000:3.0:1787757851.734494:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757851.734500:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757851.734507:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757851.734522:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655007360/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757851.734537:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757851/real 1787757851] req@ffff8c05a9f7ad80 x1874588655006976/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757851.734540:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757851.734554:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757851.734560:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757851.734573:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10908 00000100:00080000:3.0:1787757851.734581:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757851.734592:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757851.734594:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655006976/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787757851.734600:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757851.734623:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655007488/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757851.734638:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757851.734646:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757851.734653:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787757851.734658:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757851.734666:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787757851.734667:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655006976:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757851.734679:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10908 00000100:00080000:3.0:1787757851.734685:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757851.734691:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757851.734696:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757851.734712:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655007104:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757851.734726:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655007616/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757851.734743:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757851.734748:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757851.734751:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757851.734757:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10908 00000100:00080000:3.0:1787757851.734764:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787757851.734775:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757851.734777:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757851.734790:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757851.734807:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655007744/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757851.734821:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757851/real 1787757851] req@ffff8c05a9f7b800 x1874588655007104/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757851.734826:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757851.734839:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757851.734845:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757851.734860:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10908 00000100:00080000:3.0:1787757851.734867:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757851.734918:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757851.734925:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757851.734939:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655007104/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757851.734946:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655007872/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757851.734982:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757851.734993:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757851.735000:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655007104:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757851.735012:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655007232:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757851.735047:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757851.735064:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757851/real 1787757851] req@ffff8c05a9f78e00 x1874588655007232/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757851.735079:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655007232/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757851.735089:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757851.735094:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757851.735098:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655007232:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757851.735106:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655007360:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757851.735127:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757851.735139:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757851/real 1787757851] req@ffff8c05a9f79500 x1874588655007360/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757851.735153:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655007360/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757851.735163:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757851.735182:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757851.735187:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655007360:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757851.735194:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655007488:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757851.735216:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757851.735229:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757851/real 1787757851] req@ffff8c05a9f7b100 x1874588655007488/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757851.735243:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655007488/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757851.735253:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757851.735256:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757851.735260:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655007488:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757851.735309:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655007616:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757851.735339:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757851.735354:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757851/real 1787757851] req@ffff8c05a9f79180 x1874588655007616/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757851.735369:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655007616/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757851.735380:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757851.735384:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757851.735388:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655007616:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757851.735395:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655007744:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757851.735414:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757851.735426:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757851/real 1787757851] req@ffff8c05a9f79f80 x1874588655007744/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757851.735440:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655007744/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757851.735450:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757851.735453:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757851.735457:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655007744:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757851.735463:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655007872:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757851.735482:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757851.735493:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757851/real 1787757851] req@ffff8c05a9f7bb80 x1874588655007872/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757851.735507:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655007872/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757851.735525:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757851.735529:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757851.735533:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655007872:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757851.862089:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757852.310335:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757852.886046:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757853.334174:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757853.910018:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757854.358272:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757854.934063:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757855.382099:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757855.958086:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757856.406245:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787757856.854049:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757856.854068:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787757856.854074:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757856.854091:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10913 00000100:00080000:2.0:1787757856.854109:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757856.854121:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787757856.854181:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588655008000/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757856.854246:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757856.854251:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787757856.854254:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757856.854260:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10913 00000100:00080000:2.0:1787757856.854266:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757856.854272:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757856.854284:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757856.854308:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588655008128/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757856.854322:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757856.854327:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757856.854331:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757856.854337:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10913 00000100:00080000:2.0:1787757856.854341:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757856.854346:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757856.854352:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757856.854368:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655008000:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787757856.854370:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588655008256/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757856.854412:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757856.854416:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757856.854418:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757856.854427:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10913 00000100:00080000:2.0:1787757856.854436:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757856.854445:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757856.854457:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757856.854485:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588655008384/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757856.854508:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757856.854518:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:1.0:1787757856.854519:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757856.854529:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757856.854543:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10913 00000100:00080000:2.0:1787757856.854553:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757856.854560:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757856.854569:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787757856.854591:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588655008512/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757856.854603:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757856/real 1787757856] req@ffff8c05b2e70e00 x1874588655008000/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757856.854621:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757856.854645:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787757856.854654:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757856.854669:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10913 00000100:00080000:2.0:1787757856.854677:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787757856.854683:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588655008000/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757856.854690:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757856.854707:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787757856.854720:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757856.854738:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787757856.854740:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588655008640/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757856.854758:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655008000:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787757856.854768:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757856.854772:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787757856.854775:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787757856.854781:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10913 00000100:00080000:2.0:1787757856.854795:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787757856.854808:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757856.854814:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655008128:192.168.201.107@tcp:38:kworker.0 00000008:00100000:2.0:1787757856.854821:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787757856.854849:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588655008768/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787757856.854863:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787757856.854938:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787757856.854943:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787757856.854945:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787757856.854962:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10913 00000100:00080000:2.0:1787757856.854976:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787757856.854987:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757856/real 1787757856] req@ffff8c05b2e70000 x1874588655008128/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787757856.854990:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787757856.855023:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757856.855032:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588655008128/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757856.855056:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787757856.855058:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588655008896/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757856.855079:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757856.855109:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655008128:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757856.855169:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655008256:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757856.855230:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757856.855247:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757856/real 1787757856] req@ffff8c05b2e72300 x1874588655008256/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757856.855263:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588655008256/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757856.855274:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757856.855279:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757856.855283:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655008256:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757856.855292:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655008384:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757856.855316:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757856.855328:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757856/real 1787757856] req@ffff8c05b2e72a00 x1874588655008384/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757856.855343:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588655008384/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757856.855353:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757856.855357:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757856.855360:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655008384:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757856.855368:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655008512:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757856.855389:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757856.855401:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757856/real 1787757856] req@ffff8c05b2e73100 x1874588655008512/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757856.855416:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588655008512/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757856.855426:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757856.855429:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757856.855433:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655008512:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757856.855440:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655008640:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757856.855459:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757856.855471:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757856/real 1787757856] req@ffff8c05b2e72d80 x1874588655008640/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757856.855485:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588655008640/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757856.855495:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757856.855498:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757856.855502:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655008640:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757856.855508:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655008768:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757856.855527:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757856.855539:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757856/real 1787757856] req@ffff8c05b2e72680 x1874588655008768/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757856.855552:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588655008768/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757856.855563:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757856.855568:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757856.855572:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655008768:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757856.855578:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655008896:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757856.855597:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757856.855608:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757856/real 1787757856] req@ffff8c05b2e71180 x1874588655008896/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757856.855621:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588655008896/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757856.855630:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757856.855634:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757856.855637:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655008896:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757856.982048:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757857.430249:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757858.006098:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757858.454271:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757859.029985:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757859.478085:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757860.054053:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757860.502322:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757861.078016:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757861.526066:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757861.974052:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757861.974068:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787757861.974075:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757861.974092:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10918 00000100:00080000:1.0:1787757861.974142:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757861.974156:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757861.974204:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655009024/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757861.974245:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757861.974250:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757861.974253:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757861.974258:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10918 00000100:00080000:1.0:1787757861.974263:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757861.974268:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757861.974279:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757861.974295:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655009152/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757861.974309:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757861.974314:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757861.974317:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757861.974322:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10918 00000100:00080000:1.0:1787757861.974326:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757861.974332:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757861.974339:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655009024:192.168.201.107@tcp:250:kworker.0 00000008:00100000:1.0:1787757861.974341:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757861.974365:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655009280/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757861.974380:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757861.974385:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757861.974388:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757861.974393:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10918 00000100:00080000:1.0:1787757861.974398:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757861.974403:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757861.974410:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757861.974425:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655009408/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757861.974438:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787757861.974445:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757861.974447:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757861.974454:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757861.974461:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10918 00000100:00080000:1.0:1787757861.974468:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757861.974474:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757861.974480:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757861.974495:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655009536/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757861.974509:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757861.974515:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757861.974519:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757861.974526:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757861/real 1787757861] req@ffff8c05b2d9bb80 x1874588655009024/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757861.974536:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10918 00000100:00080000:1.0:1787757861.974553:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787757861.974569:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655009024/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757861.974570:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757861.974590:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787757861.974595:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757861.974606:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757861.974614:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655009024:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787757861.974624:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655009664/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757861.974628:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655009152:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787757861.974661:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757861.974666:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787757861.974669:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757861.974677:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10918 00000800:00000100:3.0:1787757861.974685:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757861.974689:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757861.974699:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757861.974707:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757861.974724:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757861/real 1787757861] req@ffff8c05b2d9aa00 x1874588655009152/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757861.974729:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655009792/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757861.974791:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757861.974796:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757861.974798:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757861.974807:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655009152/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757861.974808:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10918 00000100:00080000:1.0:1787757861.974819:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757861.974831:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757861.974845:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787757861.974848:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757861.974858:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757861.974865:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655009920/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757861.974872:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655009152:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757861.974963:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655009280:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757861.975001:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757861.975017:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757861/real 1787757861] req@ffff8c05b2d9b100 x1874588655009280/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757861.975032:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655009280/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757861.975043:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757861.975047:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757861.975051:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655009280:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757861.975059:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655009408:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757861.975078:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757861.975089:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757861/real 1787757861] req@ffff8c05b2d99880 x1874588655009408/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757861.975138:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655009408/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757861.975152:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757861.975157:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757861.975166:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655009408:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757861.975174:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655009536:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757861.975203:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757861.975223:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757861/real 1787757861] req@ffff8c05b2d98380 x1874588655009536/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757861.975237:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655009536/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757861.975248:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757861.975251:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757861.975255:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655009536:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757861.975262:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655009664:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757861.975280:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757861.975291:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757861/real 1787757861] req@ffff8c05b2d9b480 x1874588655009664/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757861.975305:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655009664/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757861.975315:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757861.975318:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757861.975322:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655009664:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757861.975328:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655009792:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757861.975347:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757861.975359:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757861/real 1787757861] req@ffff8c05b2d98000 x1874588655009792/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757861.975372:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655009792/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757861.975382:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757861.975385:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757861.975389:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655009792:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757861.975395:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655009920:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757861.975414:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757861.975425:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757861/real 1787757861] req@ffff8c05b2d99f80 x1874588655009920/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757861.975439:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655009920/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757861.975448:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757861.975452:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757861.975455:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655009920:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757862.102013:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757862.550244:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757863.125999:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757863.574138:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757864.150025:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757864.598254:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757865.174035:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757865.622145:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757866.198051:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757866.646229:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757867.093983:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757867.093996:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787757867.094002:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757867.094040:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10923 00000100:00080000:1.0:1787757867.094054:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757867.094063:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757867.094123:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655010048/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757867.094158:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757867.094163:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757867.094166:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757867.094171:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10923 00000100:00080000:1.0:1787757867.094175:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757867.094180:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757867.094191:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757867.094209:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588655010176/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757867.094225:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757867.094231:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757867.094234:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757867.094240:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10923 00000100:00080000:1.0:1787757867.094244:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787757867.094251:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655010048:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757867.094252:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757867.094266:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757867.094283:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655010304/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757867.094305:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757867.094310:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757867.094313:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757867.094319:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10923 00000100:00080000:1.0:1787757867.094323:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757867.094328:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757867.094335:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757867.094348:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655010432/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757867.094359:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757867.094363:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757867.094365:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757867.094369:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10923 00000100:00080000:1.0:1787757867.094373:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757867.094377:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757867.094382:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757867.094393:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655010560/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757867.094409:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757867.094415:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757867.094419:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757867.094426:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10923 00000100:00080000:1.0:1787757867.094435:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757867.094443:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757867.094453:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757867.094458:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655010176:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757867.094486:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655010688/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757867.094512:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655010304:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787757867.094513:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757867.094528:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787757867.094533:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757867.094543:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10923 00000100:00080000:1.0:1787757867.094552:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787757867.094561:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655010432:192.168.201.107@tcp:8:kworker.0 00000100:00080000:1.0:1787757867.094564:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757867.094581:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757867.094608:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655010560:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757867.094615:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655010816/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757867.094645:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655010688:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787757867.094660:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757867.094669:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757867.094674:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757867.094684:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655010816:192.168.201.107@tcp:8:kworker.0 00000100:00080000:1.0:1787757867.094686:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10923 00000100:00080000:1.0:1787757867.094704:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757867.094714:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757867.094726:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757867.094753:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655010944/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757867.094858:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655010944:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757867.222053:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757867.670145:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757868.246014:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757868.694263:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757869.269995:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757869.718138:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787757870.166371:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.107:988 00000400:00000100:2.0:1787757870.166425:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.107@tcp at host 192.168.201.107 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787757870.166446:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787757870.166493:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787757870.166504:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787757870.166515:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787757870.166532:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787757870.166542:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787757870.166556:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787757870.166571:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:3.0:1787757870.166579:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757867/real 1787757870] req@ffff8c05b2d98380 x1874588655010688/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757870.166618:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655010688/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757870.166636:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757870.166645:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757870.166653:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655010688:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757870.166668:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757867/real 1787757870] req@ffff8c05b2d9a300 x1874588655010176/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757870.166694:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588655010176/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757870.166710:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757870.166717:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757870.166723:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655010176:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757870.166736:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757867/real 1787757870] req@ffff8c05b2d99f80 x1874588655010304/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757870.166761:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655010304/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757870.166777:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757870.166784:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757870.166791:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655010304:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757870.166804:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757867/real 1787757870] req@ffff8c05b2d98000 x1874588655010432/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757870.166829:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655010432/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757870.166858:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757870.166865:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757870.166872:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655010432:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757870.166929:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757867/real 1787757870] req@ffff8c05b2d9b480 x1874588655010560/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757870.166982:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655010560/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757870.167000:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757870.167007:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757870.167014:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655010560:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757870.167026:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757867/real 1787757870] req@ffff8c05b2d98700 x1874588655010048/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757870.167051:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655010048/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757870.167068:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757870.167075:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757870.167081:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655010048:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757870.167131:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757867/real 1787757870] req@ffff8c05b2d9b100 x1874588655010944/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757870.167160:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655010944/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757870.167190:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757870.167200:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757870.167208:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655010944:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757870.167221:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757867/real 1787757870] req@ffff8c05b2d99880 x1874588655010816/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757870.167254:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655010816/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757870.167271:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757870.167277:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757870.167284:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655010816:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757870.294038:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757870.742241:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757871.318011:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757871.766159:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757872.214110:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757872.214126:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787757872.214134:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757872.214147:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10928 00000100:00080000:0.0:1787757872.214161:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757872.214170:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757872.214214:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655011072/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757872.214262:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757872.214267:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757872.214270:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757872.214275:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10928 00000100:00080000:0.0:1787757872.214279:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757872.214284:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757872.214296:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757872.214312:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655011200/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757872.214327:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757872.214332:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757872.214335:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757872.214345:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10928 00000100:00100000:3.0:1787757872.214345:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655011072:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757872.214355:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757872.214364:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757872.214371:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757872.214387:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588655011328/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757872.214400:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757872.214405:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757872.214407:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757872.214413:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10928 00000100:00080000:0.0:1787757872.214419:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757872.214424:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757872.214431:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757872.214446:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655011456/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787757872.214457:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757872.214479:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757872.214486:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757872.214492:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757872.214501:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10928 00000100:00080000:0.0:1787757872.214509:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757872.214518:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757872.214529:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757872.214538:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757872/real 1787757872] req@ffff8c05a9f74380 x1874588655011072/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757872.214559:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655011584/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757872.214583:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787757872.214587:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655011072/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757872.214595:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757872.214604:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757872.214623:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10928 00000100:00080000:3.0:1787757872.214630:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757872.214634:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757872.214647:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787757872.214650:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757872.214664:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655011072:192.168.201.107@tcp:250:kworker.0 00000008:00100000:0.0:1787757872.214667:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757872.214723:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588655011712/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757872.214758:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655011200:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757872.214760:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757872.214777:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787757872.214781:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787757872.214821:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757872.214830:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10928 00000100:00080000:0.0:1787757872.214845:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787757872.214857:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757872/real 1787757872] req@ffff8c05a9f74a80 x1874588655011200/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757872.214858:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757872.214975:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757872.215011:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588655011840/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757872.215025:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655011200/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757872.215041:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757872.215057:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757872.215060:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757872.215063:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757872.215077:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787757872.215085:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10928 00000100:00100000:3.0:1787757872.215097:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655011200:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757872.215100:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757872.215118:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757872.215124:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655011328:192.168.201.107@tcp:38:kworker.0 00000008:00100000:0.0:1787757872.215134:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757872.215189:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655011968/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787757872.215432:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757872.215461:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757872/real 1787757872] req@ffff8c05a9f75c00 x1874588655011328/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757872.215489:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588655011328/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757872.215507:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757872.215514:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757872.215522:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655011328:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757872.215535:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655011456:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757872.215580:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757872.215603:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757872/real 1787757872] req@ffff8c05a9f77800 x1874588655011456/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757872.215640:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655011456/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757872.215668:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757872.215675:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757872.215682:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655011456:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757872.215695:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655011584:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757872.215738:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757872.215762:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757872/real 1787757872] req@ffff8c05a9f76680 x1874588655011584/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757872.215786:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655011584/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757872.215802:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757872.215808:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757872.215814:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655011584:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757872.215855:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655011712:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757872.215961:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757872.215987:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757872/real 1787757872] req@ffff8c05a9f77b80 x1874588655011712/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757872.216011:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588655011712/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757872.216028:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757872.216034:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757872.216041:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655011712:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757872.216053:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655011840:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757872.216090:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757872.216110:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757872/real 1787757872] req@ffff8c05a9f76a00 x1874588655011840/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757872.216131:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588655011840/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757872.216160:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757872.216167:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757872.216174:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655011840:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757872.216194:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655011968:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757872.216234:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757872.216258:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757872/real 1787757872] req@ffff8c05a9f74380 x1874588655011968/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757872.216282:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655011968/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757872.216299:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757872.216306:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757872.216313:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655011968:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757872.342027:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757872.790233:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757873.366017:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757873.814194:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757874.390031:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757874.838208:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757875.413971:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757875.862112:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757876.438014:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757876.886447:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757877.333979:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757877.333992:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787757877.333998:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757877.334010:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10933 00000100:00080000:0.0:1787757877.334024:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757877.334033:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757877.334080:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655012096/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757877.334117:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757877.334122:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757877.334125:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757877.334133:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10933 00000100:00080000:0.0:1787757877.334139:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757877.334147:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757877.334163:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757877.334191:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588655012224/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757877.334210:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655012096:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757877.334223:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757877.334233:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757877.334236:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757877.334246:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10933 00000100:00080000:0.0:1787757877.334256:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757877.334264:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757877.334275:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757877.334303:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588655012352/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787757877.334321:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757877.334338:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787757877.334379:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757877/real 1787757877] req@ffff8c05a9f76300 x1874588655012096/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757877.334399:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757877.334404:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757877.334408:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655012096/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757877.334416:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10933 00000100:00080000:0.0:1787757877.334430:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757877.334437:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757877.334440:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:3.0:1787757877.334449:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000008:00100000:0.0:1787757877.334454:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757877.334457:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655012096:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757877.334496:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655012224:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757877.334502:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588655012480/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757877.334532:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787757877.334534:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757877.334547:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757877.334553:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757877.334562:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757877/real 1787757877] req@ffff8c05a9f76d80 x1874588655012224/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757877.334571:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10933 00000100:00080000:0.0:1787757877.334590:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787757877.334598:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588655012224/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757877.334609:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:3.0:1787757877.334623:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787757877.334627:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787757877.334634:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757877.334642:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655012224:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757877.334658:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655012608/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757877.334666:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655012352:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757877.334688:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757877.334694:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757877.334697:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757877.334704:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10933 00000100:00080000:0.0:1787757877.334712:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757877.334718:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757877.334727:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787757877.334737:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757877.334751:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588655012736/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757877.334771:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757877/real 1787757877] req@ffff8c05a9f74e00 x1874588655012352/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757877.334807:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757877.334814:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787757877.334821:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757877.334822:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588655012352/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757877.334854:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10933 00000100:00080000:0.0:1787757877.334865:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757877.334911:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757877.334922:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787757877.334934:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757877.334943:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757877.334958:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655012352:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757877.334960:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588655012864/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757877.334986:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787757877.334993:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655012480:192.168.201.107@tcp:8:kworker.0 00000100:00080000:0.0:1787757877.334998:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757877.335007:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757877.335018:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10933 00000100:00080000:0.0:1787757877.335029:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757877.335041:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:3.0:1787757877.335044:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:0.0:1787757877.335060:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757877.335071:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757877/real 1787757877] req@ffff8c05a9f75500 x1874588655012480/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757877.335097:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588655012992/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757877.335098:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588655012480/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757877.335158:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757877.335167:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757877.335175:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655012480:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757877.335187:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655012608:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757877.335214:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757877.335229:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757877/real 1787757877] req@ffff8c05a9f75880 x1874588655012608/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757877.335244:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655012608/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757877.335255:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757877.335258:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757877.335262:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655012608:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757877.335268:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655012736:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757877.335288:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757877.335299:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757877/real 1787757877] req@ffff8c05a9f74000 x1874588655012736/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757877.335313:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588655012736/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757877.335323:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757877.335326:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757877.335329:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655012736:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757877.335336:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655012864:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757877.335354:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757877.335365:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757877/real 1787757877] req@ffff8c05a9f75180 x1874588655012864/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757877.335379:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588655012864/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757877.335397:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757877.335401:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757877.335405:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655012864:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757877.335422:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655012992:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757877.335442:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757877.335453:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757877/real 1787757877] req@ffff8c05a9f77100 x1874588655012992/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757877.335466:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588655012992/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757877.335476:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757877.335480:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757877.335484:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655012992:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757877.462006:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757877.910136:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757878.485995:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757878.934233:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757879.509985:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757879.958091:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757880.533997:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757880.982222:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757881.558101:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757882.006167:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757882.454128:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757882.454141:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787757882.454148:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757882.454164:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10938 00000100:00080000:0.0:1787757882.454181:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757882.454194:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757882.454458:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588655013120/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757882.454510:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757882.454518:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757882.454522:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757882.454532:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10938 00000100:00080000:0.0:1787757882.454546:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757882.454558:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757882.454567:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655013120:192.168.201.107@tcp:250:kworker.0 00000008:00100000:0.0:1787757882.454575:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757882.454616:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588655013248/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757882.454633:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757882.454638:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757882.454640:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757882.454646:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10938 00000100:00080000:0.0:1787757882.454651:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757882.454656:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757882.454664:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757882.454688:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588655013376/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757882.454709:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757882.454715:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757882.454719:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757882.454727:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10938 00000100:00080000:0.0:1787757882.454736:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757882.454796:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757882.454806:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757882.454825:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655013504/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757882.454837:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757882.454842:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757882.454844:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757882.454848:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10938 00000100:00080000:0.0:1787757882.454853:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757882.454857:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757882.454862:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757882.454955:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588655013632/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757882.454970:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757882.454976:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757882.454980:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757882.454985:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10938 00000100:00080000:0.0:1787757882.454992:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757882.454997:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757882.455007:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757882.455023:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588655013760/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757882.455033:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757882.455037:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787757882.455039:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757882.455044:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10938 00000100:00080000:0.0:1787757882.455049:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757882.455052:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757882.455057:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757882.455072:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588655013888/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757882.455083:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757882.455088:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757882.455090:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757882.455094:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10938 00000100:00080000:0.0:1787757882.455100:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757882.455107:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757882.455113:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787757882.455119:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757882.455140:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655014016/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757882.455316:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757882/real 1787757882] req@ffff8c05a9f77100 x1874588655013120/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757882.455365:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588655013120/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757882.455391:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757882.455399:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757882.455405:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655013120:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757882.455445:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655013248:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757882.455482:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757882.455499:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757882/real 1787757882] req@ffff8c05a9f75180 x1874588655013248/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757882.455521:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588655013248/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757882.455539:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757882.455546:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757882.455553:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655013248:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757882.455567:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655013376:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757882.455617:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757882.455647:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757882/real 1787757882] req@ffff8c05a9f74000 x1874588655013376/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757882.455677:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588655013376/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757882.455697:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757882.455705:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757882.455718:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655013376:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757882.455767:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655013504:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757882.455807:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757882.455824:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757882/real 1787757882] req@ffff8c05a9f75880 x1874588655013504/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757882.455839:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655013504/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757882.455850:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757882.455855:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757882.455859:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655013504:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757882.455867:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655013632:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757882.455931:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757882.455954:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757882/real 1787757882] req@ffff8c05a9f75500 x1874588655013632/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757882.455971:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588655013632/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757882.455983:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757882.455988:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757882.455996:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655013632:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757882.456006:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655013760:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757882.456036:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757882.456049:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757882/real 1787757882] req@ffff8c05a9f74e00 x1874588655013760/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757882.456064:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588655013760/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757882.456074:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757882.456078:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757882.456082:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655013760:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757882.456089:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655013888:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757882.456108:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757882.456119:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757882/real 1787757882] req@ffff8c05a9f76d80 x1874588655013888/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757882.456133:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588655013888/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757882.456144:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757882.456147:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757882.456151:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655013888:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757882.456158:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655014016:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757882.456177:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757882.456189:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757882/real 1787757882] req@ffff8c05a9f76300 x1874588655014016/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757882.456203:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655014016/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757882.456213:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757882.456216:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757882.456220:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655014016:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757882.581968:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757883.030233:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757883.605991:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757884.054121:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757884.630008:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757885.078245:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757885.654015:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757886.102076:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757886.677983:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757887.126276:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757887.574063:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757887.574080:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787757887.574087:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757887.574104:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10943 00000100:00080000:1.0:1787757887.574122:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757887.574133:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757887.574200:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655014144/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757887.574251:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757887.574256:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757887.574259:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757887.574264:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10943 00000100:00080000:1.0:1787757887.574269:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757887.574273:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757887.574287:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757887.574307:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655014272/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757887.574324:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757887.574330:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757887.574332:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757887.574340:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10943 00000100:00080000:1.0:1787757887.574350:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787757887.574355:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655014144:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757887.574357:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757887.574371:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757887.574392:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655014400/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757887.574408:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757887.574412:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757887.574415:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757887.574421:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10943 00000100:00080000:1.0:1787757887.574426:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757887.574432:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757887.574439:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757887.574455:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655014528/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757887.574482:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757887.574493:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:3.0:1787757887.574496:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757887.574498:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757887.574513:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10943 00000100:00080000:1.0:1787757887.574522:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757887.574529:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757887.574537:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757887.574555:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655014656/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757887.574568:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757887.574573:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757887.574577:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757887.574588:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757887/real 1787757887] req@ffff8c05b2d9aa00 x1874588655014144/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757887.574589:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10943 00000100:00080000:1.0:1787757887.574599:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757887.574612:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757887.574622:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757887.574645:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655014784/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757887.574687:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787757887.574698:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655014144/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757887.574700:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787757887.574712:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757887.574731:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10943 00000100:00080000:1.0:1787757887.574749:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757887.574765:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:3.0:1787757887.574771:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787757887.574779:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787757887.574789:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757887.574805:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655014144:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787757887.574818:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655014912/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757887.574848:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787757887.574851:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655014272:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787757887.574870:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757887.574937:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757887.574947:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10943 00000100:00080000:1.0:1787757887.574958:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757887.574965:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:3.0:1787757887.574978:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:1.0:1787757887.574981:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757887.575024:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655015040/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757887.575026:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757887/real 1787757887] req@ffff8c05b2d9bb80 x1874588655014272/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757887.575128:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655014272/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757887.575151:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757887.575160:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757887.575167:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655014272:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757887.575182:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655014400:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757887.575218:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757887.575235:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757887/real 1787757887] req@ffff8c05b2d99880 x1874588655014400/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757887.575252:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655014400/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757887.575262:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757887.575267:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757887.575271:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655014400:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757887.575280:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655014528:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757887.575309:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757887.575330:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757887/real 1787757887] req@ffff8c05b2d9b100 x1874588655014528/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757887.575370:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655014528/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757887.575389:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757887.575396:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757887.575402:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655014528:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757887.575413:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655014656:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757887.575455:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757887.575477:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757887/real 1787757887] req@ffff8c05b2d98700 x1874588655014656/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757887.575503:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655014656/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757887.575521:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757887.575525:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757887.575530:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655014656:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757887.575538:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655014784:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757887.575568:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757887.575582:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757887/real 1787757887] req@ffff8c05b2d9b480 x1874588655014784/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757887.575602:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655014784/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757887.575630:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757887.575638:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757887.575646:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655014784:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757887.575714:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655014912:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757887.575762:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757887.575785:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757887/real 1787757887] req@ffff8c05b2d98000 x1874588655014912/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757887.575812:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655014912/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757887.575828:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757887.575833:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757887.575837:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655014912:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757887.575846:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655015040:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757887.575908:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757887.575925:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757887/real 1787757887] req@ffff8c05b2d99f80 x1874588655015040/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757887.575940:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655015040/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757887.575951:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757887.575955:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757887.575959:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655015040:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757887.702035:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757888.150102:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757888.726000:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757889.174199:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757889.750002:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757890.198168:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757890.774006:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757891.222362:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757891.798031:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757892.246148:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757892.693946:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757892.693956:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787757892.693961:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757892.693972:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10949 00000100:00080000:1.0:1787757892.693983:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757892.693991:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757892.694042:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588655015168/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757892.694071:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757892.694080:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757892.694082:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757892.694088:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10949 00000100:00080000:1.0:1787757892.694092:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757892.694097:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757892.694109:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757892.694116:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655015168:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787757892.694124:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655015296/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757892.694134:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757892.694137:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757892.694139:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757892.694143:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10949 00000100:00080000:1.0:1787757892.694145:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757892.694149:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757892.694153:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757892.694166:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655015424/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757892.694176:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757892.694179:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757892.694180:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757892.694193:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10949 00000100:00080000:1.0:1787757892.694197:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787757892.694198:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757892.694200:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757892.694205:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757892.694217:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655015552/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757892.694226:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757892.694229:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757892.694231:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757892.694235:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10949 00000100:00080000:1.0:1787757892.694238:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757892.694242:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757892.694248:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757892.694251:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757892/real 1787757892] req@ffff8c05b2d9a300 x1874588655015168/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757892.694258:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655015680/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757892.694267:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757892.694270:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757892.694272:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757892.694272:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588655015168/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757892.694276:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10949 00000100:00080000:1.0:1787757892.694279:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757892.694283:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:3.0:1787757892.694284:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787757892.694287:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787757892.694288:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757892.694292:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655015168:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787757892.694296:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655015808/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757892.694304:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757892.694307:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787757892.694309:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757892.694313:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10949 00000100:00080000:1.0:1787757892.694316:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787757892.694318:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655015296:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787757892.694320:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757892.694325:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757892.694334:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655015936/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787757892.694341:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757892.694345:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757892.694349:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757892.694351:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757892.694353:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757892/real 1787757892] req@ffff8c05b2d98380 x1874588655015296/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757892.694355:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10949 00000100:00080000:1.0:1787757892.694358:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757892.694362:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757892.694365:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655015296/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787757892.694367:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787757892.694381:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757892.694385:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757892.694386:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655016064/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757892.694389:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655015296:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757892.694395:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655015424:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757892.694415:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757892.694427:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757892/real 1787757892] req@ffff8c05b2d99f80 x1874588655015424/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757892.694440:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655015424/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757892.694446:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757892.694448:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757892.694450:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655015424:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757892.694454:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655015552:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757892.694470:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757892.694477:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757892/real 1787757892] req@ffff8c05b2d98000 x1874588655015552/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757892.694494:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655015552/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757892.694499:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757892.694501:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757892.694504:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655015552:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757892.694508:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655015680:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757892.694520:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757892.694527:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757892/real 1787757892] req@ffff8c05b2d9b480 x1874588655015680/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757892.694535:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655015680/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757892.694540:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757892.694542:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757892.694543:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655015680:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757892.694546:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655015808:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757892.694570:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757892.694581:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757892/real 1787757892] req@ffff8c05b2d98700 x1874588655015808/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757892.694590:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655015808/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757892.694595:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757892.694598:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757892.694600:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655015808:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757892.694613:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655015936:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757892.694627:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757892.694634:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757892/real 1787757892] req@ffff8c05b2d9b100 x1874588655015936/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757892.694641:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655015936/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757892.694647:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757892.694648:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757892.694650:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655015936:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757892.694653:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655016064:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757892.694669:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757892.694680:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757892/real 1787757892] req@ffff8c05b2d99880 x1874588655016064/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757892.694693:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655016064/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757892.694703:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757892.694707:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757892.694710:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655016064:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757892.822014:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757893.270265:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757893.846033:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757894.294122:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757894.870027:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757895.318275:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757895.894074:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757896.342140:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757896.917988:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757897.366223:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757897.814026:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757897.814034:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787757897.814037:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757897.814046:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10954 00000100:00080000:0.0:1787757897.814055:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757897.814061:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757897.814093:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655016192/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757897.814118:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757897.814121:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757897.814122:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757897.814126:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10954 00000100:00080000:0.0:1787757897.814129:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757897.814131:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757897.814145:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757897.814156:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588655016320/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757897.814166:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757897.814169:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757897.814172:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757897.814177:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655016192:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757897.814181:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10954 00000100:00080000:0.0:1787757897.814193:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757897.814198:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757897.814205:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757897.814223:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588655016448/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757897.814232:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757897.814235:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757897.814237:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757897.814241:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10954 00000100:00080000:0.0:1787757897.814244:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757897.814247:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757897.814251:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757897.814262:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655016576/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787757897.814273:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757897.814279:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757897.814282:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757897.814284:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757897.814287:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10954 00000100:00080000:0.0:1787757897.814290:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757897.814293:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757897.814297:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757897.814304:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655016704/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757897.814315:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757897.814318:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757897.814319:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757897.814322:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10954 00000100:00080000:0.0:1787757897.814325:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757897.814327:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757897.814330:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757897.814337:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588655016832/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757897.814342:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757897.814345:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787757897.814347:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757897.814352:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757897/real 1787757897] req@ffff8c05a9f74380 x1874588655016192/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757897.814353:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10954 00000100:00080000:0.0:1787757897.814363:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757897.814377:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757897.814387:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757897.814398:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655016960/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757897.814401:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655016192/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757897.814426:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757897.814428:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757897.814439:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787757897.814440:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757897.814448:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757897.814456:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655016192:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757897.814460:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10954 00000100:00080000:0.0:1787757897.814488:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757897.814493:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757897.814499:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757897.814520:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655016320:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757897.814527:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655017088/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787757897.814551:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757897.814566:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757897/real 1787757897] req@ffff8c05a9f76a00 x1874588655016320/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757897.814576:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588655016320/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757897.814583:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757897.814585:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757897.814588:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655016320:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757897.814593:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655016448:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757897.814605:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757897.814612:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757897/real 1787757897] req@ffff8c05a9f77b80 x1874588655016448/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757897.814624:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588655016448/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757897.814630:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757897.814631:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757897.814633:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655016448:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757897.814637:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655016576:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757897.814649:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757897.814656:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757897/real 1787757897] req@ffff8c05a9f76680 x1874588655016576/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757897.814664:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655016576/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757897.814669:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757897.814671:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757897.814672:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655016576:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757897.814677:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655016704:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757897.814688:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757897.814694:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757897/real 1787757897] req@ffff8c05a9f77800 x1874588655016704/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757897.814701:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655016704/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757897.814711:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757897.814714:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757897.814716:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655016704:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757897.814719:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655016832:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757897.814730:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757897.814735:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757897/real 1787757897] req@ffff8c05a9f75c00 x1874588655016832/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757897.814742:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588655016832/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757897.814748:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757897.814750:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757897.814752:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655016832:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757897.814755:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655016960:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757897.814765:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757897.814771:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757897/real 1787757897] req@ffff8c05a9f74a80 x1874588655016960/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757897.814778:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655016960/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757897.814783:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757897.814785:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757897.814788:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655016960:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757897.814803:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655017088:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757897.814814:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757897.814820:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757897/real 1787757897] req@ffff8c05a9f74380 x1874588655017088/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757897.814828:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655017088/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757897.814833:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757897.814835:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757897.814837:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655017088:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757897.941966:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757898.390020:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757898.965993:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757899.414121:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757899.990019:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757900.438141:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757901.013999:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757901.462249:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757902.038003:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757902.486157:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757902.933982:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757902.933995:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787757902.934000:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757902.934013:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10959 00000100:00080000:0.0:1787757902.934027:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757902.934036:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757902.934085:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655017216/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757902.934126:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757902.934131:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757902.934133:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757902.934139:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10959 00000100:00080000:0.0:1787757902.934155:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757902.934161:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757902.934172:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757902.934187:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588655017344/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757902.934202:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757902.934209:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757902.934212:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757902.934217:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10959 00000100:00080000:0.0:1787757902.934222:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757902.934226:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757902.934232:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757902.934247:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588655017472/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757902.934271:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655017216:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757902.934275:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757902.934298:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757902.934302:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757902.934327:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10959 00000100:00080000:0.0:1787757902.934338:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757902.934348:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757902.934359:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757902.934415:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588655017600/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757902.934433:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757902.934445:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:3.0:1787757902.934446:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757902.934454:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757902.934464:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10959 00000100:00080000:0.0:1787757902.934473:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757902.934480:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757902.934493:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757902.934514:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655017728/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757902.934529:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787757902.934535:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757902/real 1787757902] req@ffff8c05a9f76300 x1874588655017216/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757902.934538:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757902.934542:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757902.934568:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10959 00000100:00080000:0.0:1787757902.934586:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757902.934595:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757902.934604:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757902.934614:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655017216/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757902.934629:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588655017856/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757902.934638:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757902.934653:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757902.934663:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787757902.934670:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787757902.934673:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757902.934679:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10959 00000100:00080000:0.0:1787757902.934682:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757902.934697:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757902.934699:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655017216:192.168.201.107@tcp:250:kworker.0 00000008:00100000:0.0:1787757902.934707:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757902.934721:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655017344:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757902.934730:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588655017984/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757902.934753:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757902.934757:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757902.934762:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757902.934772:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10959 00000800:00000100:3.0:1787757902.934777:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757902.934793:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787757902.934805:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757902/real 1787757902] req@ffff8c05a9f76d80 x1874588655017344/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757902.934811:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757902.934829:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757902.934852:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588655017344/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757902.934861:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588655018112/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757902.934938:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757902.934952:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757902.934963:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655017344:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757902.935007:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655017472:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757902.935037:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757902.935053:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757902/real 1787757902] req@ffff8c05a9f74e00 x1874588655017472/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757902.935069:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588655017472/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757902.935079:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757902.935084:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757902.935088:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655017472:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757902.935096:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655017600:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757902.935118:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757902.935130:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757902/real 1787757902] req@ffff8c05a9f75500 x1874588655017600/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757902.935145:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588655017600/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757902.935155:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757902.935159:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757902.935163:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655017600:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757902.935170:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655017728:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757902.935189:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757902.935201:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757902/real 1787757902] req@ffff8c05a9f75880 x1874588655017728/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757902.935215:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655017728/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757902.935225:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757902.935229:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757902.935232:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655017728:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757902.935239:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655017856:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757902.935259:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757902.935270:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757902/real 1787757902] req@ffff8c05a9f74000 x1874588655017856/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757902.935292:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588655017856/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757902.935302:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757902.935306:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757902.935310:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655017856:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757902.935316:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655017984:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757902.935335:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757902.935347:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757902/real 1787757902] req@ffff8c05a9f75180 x1874588655017984/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757902.935361:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588655017984/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757902.935372:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757902.935375:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757902.935405:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655017984:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757902.935413:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655018112:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757902.935436:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757902.935449:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757902/real 1787757902] req@ffff8c05a9f77100 x1874588655018112/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757902.935464:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588655018112/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757902.935482:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757902.935487:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757902.935491:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655018112:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757903.062011:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757903.510227:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757904.086004:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757904.534203:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757905.110014:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757905.558307:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757906.134021:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757906.582140:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757907.158008:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757907.606238:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757908.054064:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757908.054078:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757908.054084:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757908.054097:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10964 00000100:00080000:3.0:1787757908.054113:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757908.054126:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757908.054184:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655018240/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757908.054233:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757908.054240:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757908.054244:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757908.054253:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10964 00000100:00080000:3.0:1787757908.054262:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757908.054273:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757908.054326:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655018240:192.168.201.107@tcp:250:kworker.0 00000008:00100000:3.0:1787757908.054327:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757908.054362:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655018368/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757908.054381:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757908.054387:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757908.054390:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757908.054395:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10964 00000100:00080000:3.0:1787757908.054402:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757908.054408:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757908.054415:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757908.054430:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655018496/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757908.054442:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757908.054446:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757908.054448:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757908.054452:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10964 00000100:00080000:3.0:1787757908.054456:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757908.054460:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757908.054465:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757908.054476:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655018624/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787757908.054484:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757908.054501:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757908.054506:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757908.054509:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757908.054516:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10964 00000100:00080000:3.0:1787757908.054523:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757908.054529:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757908.054536:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757908.054551:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655018752/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757908.054564:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757908.054568:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757908.054575:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757908.054581:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10964 00000100:00080000:3.0:1787757908.054585:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757908.054589:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757908.054592:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757908/real 1787757908] req@ffff8c05a9f7bb80 x1874588655018240/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787757908.054595:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757908.054617:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655018880/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757908.054642:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757908.054646:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757908.054649:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757908.054657:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10964 00000100:00100000:1.0:1787757908.054660:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655018240/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757908.054662:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757908.054673:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757908.054686:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787757908.054699:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787757908.054708:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655019008/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757908.054714:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757908.054731:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655018240:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757908.054739:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757908.054745:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757908.054748:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757908.054755:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10964 00000100:00080000:3.0:1787757908.054759:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757908.054764:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757908.054773:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757908.054775:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655018368:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757908.054809:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655019136/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787757908.054832:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757908.054861:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757908/real 1787757908] req@ffff8c05a9f79f80 x1874588655018368/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757908.054932:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655018368/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757908.054945:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757908.054949:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757908.054954:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655018368:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757908.054962:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655018496:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757908.054991:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757908.055014:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757908/real 1787757908] req@ffff8c05a9f79180 x1874588655018496/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757908.055038:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655018496/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757908.055055:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757908.055062:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757908.055069:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655018496:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757908.055082:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655018624:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757908.055120:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757908.055137:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757908/real 1787757908] req@ffff8c05a9f7b100 x1874588655018624/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757908.055153:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655018624/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757908.055164:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757908.055168:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757908.055172:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655018624:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757908.055179:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655018752:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757908.055201:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757908.055213:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757908/real 1787757908] req@ffff8c05a9f79500 x1874588655018752/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757908.055227:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655018752/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757908.055237:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757908.055241:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757908.055244:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655018752:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757908.055250:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655018880:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757908.055269:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757908.055281:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757908/real 1787757908] req@ffff8c05a9f78e00 x1874588655018880/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757908.055331:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655018880/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757908.055343:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757908.055347:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757908.055351:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655018880:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757908.055359:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655019008:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757908.055386:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757908.055399:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757908/real 1787757908] req@ffff8c05a9f7b800 x1874588655019008/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757908.055413:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655019008/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757908.055423:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757908.055427:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757908.055431:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655019008:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757908.055457:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655019136:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757908.055478:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757908.055490:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757908/real 1787757908] req@ffff8c05a9f7ad80 x1874588655019136/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757908.055504:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655019136/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757908.055514:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757908.055517:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757908.055521:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655019136:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757908.182054:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757908.630223:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757909.205981:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757909.654253:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757910.230011:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757910.678232:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757911.254055:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757911.702318:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787757912.277977:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757912.726183:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757913.174092:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757913.174104:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787757913.174109:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757913.174122:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10969 00000100:00080000:0.0:1787757913.174136:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757913.174145:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757913.174249:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588655019264/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757913.174304:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757913.174311:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757913.174315:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757913.174324:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10969 00000100:00080000:0.0:1787757913.174333:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757913.174344:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757913.174357:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757913.174382:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588655019392/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757913.174404:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655019264:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757913.174410:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757913.174434:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757913.174440:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757913.174446:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10969 00000100:00080000:0.0:1787757913.174450:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757913.174455:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757913.174461:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757913.174476:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588655019520/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757913.174490:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757913.174495:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757913.174497:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757913.174503:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10969 00000100:00080000:0.0:1787757913.174506:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757913.174511:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757913.174517:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757913.174531:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655019648/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757913.174545:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757913.174550:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757913.174552:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757913.174557:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10969 00000100:00080000:0.0:1787757913.174574:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757913.174579:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:1.0:1787757913.174590:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:0.0:1787757913.174591:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757913.174617:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588655019776/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757913.174631:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757913.174635:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757913.174637:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757913.174643:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10969 00000100:00080000:0.0:1787757913.174650:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757913.174655:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757913.174662:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757913.174673:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588655019904/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757913.174686:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757913.174691:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787757913.174693:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757913.174703:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10969 00000100:00080000:0.0:1787757913.174708:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757913.174714:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757913.174724:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757913.174725:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757913/real 1787757913] req@ffff8c05a9f77100 x1874588655019264/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757913.174754:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588655020032/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757913.174772:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787757913.174778:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588655019264/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757913.174780:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757913.174790:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757913.174812:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10969 00000100:00080000:0.0:1787757913.174816:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757913.174820:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757913.174826:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757913.174846:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655020160/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757913.174852:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757913.174872:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757913.174970:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655019264:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787757913.175016:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655019392:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757913.175060:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757913.175080:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757913/real 1787757913] req@ffff8c05a9f75180 x1874588655019392/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757913.175105:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588655019392/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757913.175122:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757913.175129:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757913.175136:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655019392:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757913.175150:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655019520:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757913.175249:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757913.175270:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757913/real 1787757913] req@ffff8c05a9f74000 x1874588655019520/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757913.175287:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588655019520/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757913.175299:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757913.175305:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757913.175310:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655019520:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757913.175320:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655019648:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757913.175349:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757913.175362:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757913/real 1787757913] req@ffff8c05a9f75880 x1874588655019648/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757913.175377:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655019648/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757913.175388:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757913.175393:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757913.175397:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655019648:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757913.175404:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655019776:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757913.175429:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757913.175449:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757913/real 1787757913] req@ffff8c05a9f75500 x1874588655019776/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757913.175488:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588655019776/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757913.175507:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757913.175514:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757913.175521:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655019776:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757913.175535:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655019904:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757913.175566:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757913.175582:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757913/real 1787757913] req@ffff8c05a9f74e00 x1874588655019904/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757913.175599:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588655019904/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757913.175615:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757913.175622:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757913.175629:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655019904:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757913.175642:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655020032:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757913.175684:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757913.175707:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757913/real 1787757913] req@ffff8c05a9f76d80 x1874588655020032/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757913.175724:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588655020032/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757913.175743:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757913.175748:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757913.175752:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655020032:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757913.175760:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655020160:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757913.175787:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757913.175799:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757913/real 1787757913] req@ffff8c05a9f76300 x1874588655020160/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757913.175816:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655020160/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757913.175832:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757913.175839:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757913.175846:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655020160:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757913.302074:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757913.750311:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757914.326038:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757914.774164:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757915.350032:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757915.798249:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757916.374024:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757916.822174:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757917.397985:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757917.846332:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757918.294037:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757918.294053:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787757918.294060:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757918.294076:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10974 00000100:00080000:0.0:1787757918.294094:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757918.294105:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757918.294197:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655020288/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757918.294241:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757918.294246:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757918.294249:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757918.294255:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10974 00000100:00080000:0.0:1787757918.294260:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757918.294265:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757918.294281:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757918.294300:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655020416/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757918.294314:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757918.294320:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757918.294322:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757918.294327:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10974 00000100:00080000:0.0:1787757918.294331:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757918.294335:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757918.294344:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757918.294346:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655020288:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787757918.294362:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588655020544/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757918.294379:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757918.294384:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757918.294386:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757918.294391:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10974 00000100:00080000:0.0:1787757918.294395:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757918.294399:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757918.294406:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757918.294420:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655020672/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757918.294432:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757918.294437:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757918.294440:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757918.294445:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10974 00000100:00080000:0.0:1787757918.294462:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757918.294468:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757918.294475:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787757918.294483:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757918.294496:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655020800/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757918.294511:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757918.294516:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757918.294518:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757918.294524:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10974 00000100:00080000:0.0:1787757918.294529:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757918.294533:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757918.294539:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757918.294556:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588655020928/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757918.294570:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757918/real 1787757918] req@ffff8c05a9f74380 x1874588655020288/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757918.294574:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757918.294587:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787757918.294593:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757918.294609:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10974 00000100:00080000:0.0:1787757918.294617:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757918.294623:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757918.294629:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757918.294632:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655020288/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757918.294649:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588655021056/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757918.294703:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757918.294705:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757918.294709:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757918.294715:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757918.294719:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787757918.294727:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10974 00000100:00100000:1.0:1787757918.294736:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655020288:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757918.294737:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757918.294750:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757918.294758:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655020416:192.168.201.107@tcp:38:kworker.0 00000008:00100000:0.0:1787757918.294759:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757918.294793:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655021184/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787757918.294812:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757918.294840:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757918/real 1787757918] req@ffff8c05a9f74a80 x1874588655020416/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757918.294858:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655020416/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757918.294870:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757918.294914:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757918.294921:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655020416:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757918.294961:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655020544:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757918.294990:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757918.295010:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757918/real 1787757918] req@ffff8c05a9f75c00 x1874588655020544/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757918.295031:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588655020544/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757918.295049:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757918.295054:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757918.295061:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655020544:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757918.295071:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655020672:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757918.295160:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757918.295179:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757918/real 1787757918] req@ffff8c05a9f77800 x1874588655020672/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757918.295200:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655020672/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757918.295214:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757918.295222:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757918.295230:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655020672:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757918.295242:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655020800:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757918.295297:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757918.295330:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757918/real 1787757918] req@ffff8c05a9f76680 x1874588655020800/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757918.295357:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655020800/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757918.295376:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757918.295383:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757918.295390:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655020800:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757918.295404:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655020928:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757918.295454:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757918.295482:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757918/real 1787757918] req@ffff8c05a9f77b80 x1874588655020928/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757918.295504:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588655020928/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757918.295518:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757918.295522:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757918.295527:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655020928:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757918.295534:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655021056:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757918.295554:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757918.295567:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757918/real 1787757918] req@ffff8c05a9f76a00 x1874588655021056/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757918.295583:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588655021056/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757918.295594:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757918.295598:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757918.295602:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655021056:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757918.295609:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655021184:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757918.295632:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757918.295651:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757918/real 1787757918] req@ffff8c05a9f76300 x1874588655021184/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757918.295667:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655021184/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757918.295686:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757918.295690:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757918.295694:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655021184:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757918.422073:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757918.870191:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757919.446009:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757919.894264:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757920.470029:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757920.918160:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757921.493931:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757921.942160:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757922.517996:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757922.966127:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757923.414098:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757923.414111:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757923.414117:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757923.414129:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10979 00000100:00080000:3.0:1787757923.414142:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757923.414152:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757923.414198:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655021312/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757923.414242:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757923.414247:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757923.414250:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757923.414255:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10979 00000100:00080000:3.0:1787757923.414259:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757923.414273:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757923.414284:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757923.414300:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655021440/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757923.414315:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757923.414320:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757923.414322:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757923.414328:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10979 00000100:00080000:3.0:1787757923.414332:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757923.414337:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757923.414344:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757923.414357:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655021312:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757923.414359:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655021568/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757923.414426:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757923.414435:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757923.414439:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757923.414447:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10979 00000800:00000100:1.0:1787757923.414458:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757923.414461:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757923.414475:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757923.414488:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757923.414521:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655021696/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757923.414543:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787757923.414546:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757923/real 1787757923] req@ffff8c05a9f7b480 x1874588655021312/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757923.414550:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757923.414555:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757923.414569:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10979 00000100:00080000:3.0:1787757923.414579:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757923.414591:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757923.414600:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757923.414610:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655021312/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757923.414622:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655021824/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757923.414638:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757923.414650:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757923.414654:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787757923.414660:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757923.414667:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787757923.414673:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655021312:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757923.414677:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10979 00000100:00100000:1.0:1787757923.414690:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655021440:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757923.414701:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757923.414708:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757923.414717:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787757923.414735:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757923.414738:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655021952/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757923.414755:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787757923.414764:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757923/real 1787757923] req@ffff8c05a9f78380 x1874588655021440/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757923.414764:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757923.414772:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757923.414787:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10979 00000100:00080000:3.0:1787757923.414797:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787757923.414804:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655021440/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757923.414806:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757923.414820:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787757923.414826:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757923.414835:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757923.414841:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655022080/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757923.414849:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655021440:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757923.414914:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757923.414920:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757923.414923:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757923.414931:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10979 00000100:00080000:3.0:1787757923.414937:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757923.414943:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757923.414950:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757923.414968:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655022208/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757923.414989:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655021568:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757923.415063:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757923.415085:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757923/real 1787757923] req@ffff8c05a9f78a80 x1874588655021568/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757923.415101:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655021568/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757923.415112:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757923.415116:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757923.415120:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655021568:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757923.415128:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655021696:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757923.415151:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757923.415163:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757923/real 1787757923] req@ffff8c05a9f7a680 x1874588655021696/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757923.415177:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655021696/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757923.415186:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757923.415190:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757923.415193:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655021696:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757923.415200:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655021824:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757923.415218:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757923.415230:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757923/real 1787757923] req@ffff8c05a9f7aa00 x1874588655021824/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757923.415244:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655021824/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757923.415254:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757923.415257:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757923.415260:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655021824:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757923.415267:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655021952:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757923.415284:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757923.415295:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757923/real 1787757923] req@ffff8c05a9f7a300 x1874588655021952/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757923.415308:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655021952/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757923.415318:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757923.415321:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757923.415324:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655021952:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757923.415330:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655022080:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757923.415348:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757923.415358:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757923/real 1787757923] req@ffff8c05a9f79880 x1874588655022080/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757923.415371:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655022080/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757923.415381:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757923.415385:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757923.415389:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655022080:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757923.415409:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655022208:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757923.415428:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757923.415439:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757923/real 1787757923] req@ffff8c05a9f78700 x1874588655022208/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757923.415456:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655022208/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757923.415473:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757923.415479:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757923.415485:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655022208:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757923.542056:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757923.990275:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757924.566062:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757925.014128:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757925.590050:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757926.038209:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757926.614063:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757927.062137:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757927.638051:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757928.086326:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757928.534183:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757928.534196:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787757928.534201:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757928.534214:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10984 00000100:00080000:0.0:1787757928.534227:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757928.534235:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757928.534284:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588655022336/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757928.534325:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757928.534330:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757928.534333:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757928.534339:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10984 00000100:00080000:0.0:1787757928.534343:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757928.534348:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757928.534359:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757928.534377:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588655022464/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757928.534393:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757928.534397:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757928.534403:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787757928.534407:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655022336:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757928.534413:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10984 00000100:00080000:0.0:1787757928.534427:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757928.534431:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757928.534438:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757928.534455:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588655022592/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757928.534467:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757928.534472:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757928.534475:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757928.534481:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10984 00000100:00080000:0.0:1787757928.534497:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757928.534501:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757928.534509:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787757928.534511:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757928.534533:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655022720/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757928.534551:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757928.534558:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757928.534562:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787757928.534574:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757928/real 1787757928] req@ffff8c05a9f76d80 x1874588655022336/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757928.534575:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10984 00000100:00080000:0.0:1787757928.534591:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757928.534606:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757928.534615:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588655022336/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787757928.534622:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787757928.534635:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757928.534643:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757928.534652:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655022336:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787757928.534659:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588655022848/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757928.534686:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787757928.534688:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655022464:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757928.534700:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757928.534706:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757928.534713:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10984 00000800:00000100:1.0:1787757928.534732:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757928.534734:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757928.534746:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757928.534756:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757928.534757:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757928/real 1787757928] req@ffff8c05a9f74e00 x1874588655022464/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757928.534783:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588655022976/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757928.534786:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588655022464/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757928.534837:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757928.534840:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757928.534847:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787757928.534853:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787757928.534855:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787757928.534866:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655022464:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757928.534922:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10984 00000100:00080000:0.0:1787757928.534958:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757928.534967:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757928.534983:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655022592:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757928.535035:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757928.535059:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757928/real 1787757928] req@ffff8c05a9f75500 x1874588655022592/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787757928.535074:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757928.535086:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588655022592/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757928.535099:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787757928.535101:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588655023104/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757928.535112:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757928.535127:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655022592:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757928.535140:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787757928.535148:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655022720:192.168.201.107@tcp:8:kworker.0 00000100:00080000:0.0:1787757928.535150:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757928.535158:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757928.535168:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10984 00000100:00080000:0.0:1787757928.535176:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757928.535181:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757928.535189:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787757928.535198:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757928.535216:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588655023232/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757928.535224:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757928/real 1787757928] req@ffff8c05a9f75880 x1874588655022720/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757928.535263:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655022720/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757928.535274:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757928.535280:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757928.535284:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655022720:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757928.535308:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655022848:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757928.535331:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757928.535344:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757928/real 1787757928] req@ffff8c05a9f74000 x1874588655022848/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757928.535358:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588655022848/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757928.535368:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757928.535371:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757928.535375:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655022848:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757928.535382:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655022976:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757928.535399:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757928.535410:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757928/real 1787757928] req@ffff8c05a9f75180 x1874588655022976/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757928.535433:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588655022976/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757928.535442:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757928.535446:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757928.535449:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655022976:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757928.535456:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655023104:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757928.535474:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757928.535485:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757928/real 1787757928] req@ffff8c05a9f77100 x1874588655023104/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757928.535499:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588655023104/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757928.535508:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757928.535511:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757928.535515:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655023104:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757928.535521:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655023232:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757928.535539:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757928.535549:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757928/real 1787757928] req@ffff8c05a9f76d80 x1874588655023232/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757928.535562:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588655023232/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757928.535580:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757928.535583:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757928.535587:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655023232:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757928.662032:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757929.110121:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757929.686018:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757930.134187:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757930.710045:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757931.158124:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757931.734036:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757932.182244:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757932.758013:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757933.206113:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757933.654091:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757933.654107:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787757933.654114:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757933.654129:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10990 00000100:00080000:1.0:1787757933.654147:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757933.654158:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757933.654220:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655023360/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757933.654268:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757933.654274:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757933.654276:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757933.654296:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10990 00000100:00080000:1.0:1787757933.654301:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757933.654307:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757933.654321:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757933.654339:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655023488/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757933.654354:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757933.654361:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757933.654364:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757933.654369:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10990 00000100:00080000:1.0:1787757933.654374:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757933.654378:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757933.654386:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757933.654397:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655023360:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787757933.654407:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655023616/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757933.654424:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757933.654428:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757933.654431:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757933.654436:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10990 00000100:00080000:1.0:1787757933.654441:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757933.654445:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757933.654452:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757933.654477:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655023744/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757933.654497:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757933.654505:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757933.654509:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757933.654516:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10990 00000100:00080000:1.0:1787757933.654525:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757933.654535:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757933.654547:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757933.654574:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655023872/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757933.654589:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655023488:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787757933.654600:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757933.654608:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757933.654611:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757933.654619:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10990 00000100:00080000:1.0:1787757933.654626:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757933.654632:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757933.654642:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757933.654668:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655024000/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757933.654678:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655023616:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787757933.654696:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757933.654703:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787757933.654706:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757933.654716:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10990 00000100:00080000:1.0:1787757933.654722:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787757933.654727:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655023744:192.168.201.107@tcp:8:kworker.0 00000100:00080000:1.0:1787757933.654734:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757933.654752:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757933.654780:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655024128/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757933.654793:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757933.654798:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757933.654801:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757933.654808:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10990 00000100:00080000:1.0:1787757933.654814:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787757933.654820:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655023872:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787757933.654825:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757933.654869:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757933.654951:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655024256/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757933.654990:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655024000:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757933.655018:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655024128:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757933.655040:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655024256:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757933.781981:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757934.230214:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757934.806001:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757935.254104:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757935.830023:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757936.278256:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787757936.726236:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.107:988 00000400:00000100:0.0:1787757936.726301:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.107@tcp at host 192.168.201.107 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787757936.726322:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787757936.726369:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787757936.726380:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787757936.726390:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787757936.726401:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787757936.726412:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787757936.726421:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787757936.726432:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:3.0:1787757936.726478:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757933/real 1787757936] req@ffff8c05b2d9b480 x1874588655024000/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757936.726520:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655024000/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757936.726544:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757936.726552:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757936.726562:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655024000:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757936.726578:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757933/real 1787757936] req@ffff8c05b2d98000 x1874588655024128/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757936.726604:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655024128/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757936.726619:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757936.726626:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757936.726633:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655024128:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757936.726647:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757933/real 1787757936] req@ffff8c05b2d99f80 x1874588655024256/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757936.726670:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655024256/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757936.726687:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757936.726693:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757936.726700:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655024256:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757936.726713:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757933/real 1787757936] req@ffff8c05b2d99880 x1874588655023616/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757936.726737:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655023616/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757936.726753:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757936.726760:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757936.726845:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655023616:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757936.726861:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757933/real 1787757936] req@ffff8c05b2d9b100 x1874588655023744/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757936.726946:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655023744/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757936.726965:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757936.726972:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757936.726979:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655023744:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757936.726994:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757933/real 1787757936] req@ffff8c05b2d98700 x1874588655023872/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757936.727019:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655023872/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757936.727035:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757936.727041:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757936.727048:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655023872:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757936.727061:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757933/real 1787757936] req@ffff8c05b2d9bb80 x1874588655023360/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757936.727084:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655023360/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757936.727100:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757936.727106:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757936.727114:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655023360:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757936.727127:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757933/real 1787757936] req@ffff8c05b2d9aa00 x1874588655023488/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757936.727151:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655023488/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757936.727168:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757936.727175:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757936.727182:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655023488:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787757936.854070:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757937.302155:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757937.877996:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757938.326171:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757938.774025:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757938.774042:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757938.774050:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757938.774067:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 10995 00000100:00080000:3.0:1787757938.774084:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757938.774097:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757938.774151:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655024384/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757938.774193:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757938.774200:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757938.774204:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757938.774212:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10995 00000100:00080000:3.0:1787757938.774221:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757938.774230:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757938.774246:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757938.774260:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655024384:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757938.774271:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655024512/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757938.774290:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757938.774296:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757938.774299:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757938.774305:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10995 00000100:00080000:3.0:1787757938.774311:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757938.774317:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757938.774325:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757938.774343:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655024640/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787757938.774349:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757938.774362:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757938.774367:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757938.774371:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757938.774377:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 10995 00000100:00080000:3.0:1787757938.774383:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757938.774390:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757938.774399:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757938.774407:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757938/real 1787757938] req@ffff8c05a9f78700 x1874588655024384/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757938.774416:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655024768/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757938.774434:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787757938.774440:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655024384/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757938.774442:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757938.774445:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757938.774453:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10995 00000100:00080000:3.0:1787757938.774458:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757938.774461:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757938.774464:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:1.0:1787757938.774468:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000008:00100000:3.0:1787757938.774472:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757938.774476:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655024384:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757938.774488:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655024896/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757938.774504:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787757938.774510:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655024512:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757938.774510:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757938.774514:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757938.774520:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10995 00000100:00080000:3.0:1787757938.774526:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757938.774533:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757938.774542:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757938.774561:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655025024/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787757938.774567:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757938.774581:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757938.774587:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787757938.774590:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757938/real 1787757938] req@ffff8c05a9f79880 x1874588655024512/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757938.774591:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757938.774598:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10995 00000100:00080000:3.0:1787757938.774604:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757938.774611:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757938.774616:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655024512/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787757938.774621:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757938.774639:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655025152/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757938.774659:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757938.774661:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757938.774665:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757938.774668:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787757938.774669:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787757938.774676:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655024512:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757938.774676:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 10995 00000100:00080000:3.0:1787757938.774682:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787757938.774688:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655024640:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757938.774689:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757938.774697:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757938.774714:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655025280/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787757938.774725:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757938.774746:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757938/real 1787757938] req@ffff8c05a9f7a300 x1874588655024640/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757938.774806:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655024640/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757938.774824:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757938.774831:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757938.774838:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655024640:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757938.774850:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655024768:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757938.774935:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757938.774962:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757938/real 1787757938] req@ffff8c05a9f7aa00 x1874588655024768/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757938.774984:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655024768/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757938.775001:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757938.775008:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757938.775016:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655024768:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757938.775027:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655024896:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757938.775068:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757938.775092:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757938/real 1787757938] req@ffff8c05a9f7a680 x1874588655024896/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757938.775117:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655024896/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757938.775134:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757938.775140:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757938.775147:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655024896:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757938.775182:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655025024:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757938.775223:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757938.775245:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757938/real 1787757938] req@ffff8c05a9f78a80 x1874588655025024/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757938.775268:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655025024/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757938.775284:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757938.775291:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757938.775296:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655025024:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757938.775309:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655025152:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757938.775348:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757938.775373:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757938/real 1787757938] req@ffff8c05a9f78380 x1874588655025152/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757938.775398:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655025152/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757938.775414:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757938.775419:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757938.775426:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655025152:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757938.775440:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655025280:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757938.775480:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757938.775504:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757938/real 1787757938] req@ffff8c05a9f7b480 x1874588655025280/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757938.775528:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655025280/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757938.775545:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757938.775551:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757938.775558:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655025280:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757938.902026:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757939.350100:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757939.926035:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757940.374288:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757940.949998:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757941.398135:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757941.973993:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757942.422275:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757942.998016:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757943.446148:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757943.894031:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757943.894047:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757943.894054:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757943.894070:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11000 00000100:00080000:3.0:1787757943.894088:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757943.894099:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757943.894166:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655025408/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757943.894210:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757943.894216:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757943.894219:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757943.894225:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11000 00000100:00080000:3.0:1787757943.894230:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757943.894235:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757943.894250:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757943.894265:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655025536/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757943.894279:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757943.894283:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757943.894285:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757943.894290:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11000 00000100:00080000:3.0:1787757943.894295:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757943.894299:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757943.894305:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757943.894319:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655025664/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757943.894332:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757943.894337:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757943.894339:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757943.894345:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11000 00000100:00080000:3.0:1787757943.894349:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757943.894354:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757943.894360:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757943.894378:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655025408:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757943.894381:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655025792/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757943.894431:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757943.894436:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757943.894440:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757943.894446:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11000 00000100:00080000:3.0:1787757943.894452:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757943.894457:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757943.894463:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757943.894484:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655025920/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757943.894496:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757943.894502:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757943.894505:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757943.894510:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11000 00000100:00080000:3.0:1787757943.894517:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787757943.894532:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757943.894539:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757943.894550:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757943.894575:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655026048/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757943.894587:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757943.894592:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757943.894594:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757943.894600:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11000 00000100:00080000:3.0:1787757943.894604:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757943.894611:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757943.894618:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757943.894641:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655026176/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757943.894646:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757943/real 1787757943] req@ffff8c05a9f7ad80 x1874588655025408/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757943.894688:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757943.894707:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757943.894715:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757943.894728:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11000 00000100:00080000:3.0:1787757943.894738:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757943.894749:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757943.894761:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757943.894786:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655026304/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757943.894810:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655025408/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757943.894830:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757943.894837:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757943.894843:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655025408:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787757943.894856:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655025536:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757943.894951:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757943.894971:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757943/real 1787757943] req@ffff8c05a9f7b800 x1874588655025536/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757943.894988:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655025536/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757943.894999:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757943.895003:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757943.895008:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655025536:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757943.895016:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655025664:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757943.895038:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757943.895050:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757943/real 1787757943] req@ffff8c05a9f78e00 x1874588655025664/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757943.895065:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655025664/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757943.895082:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757943.895086:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757943.895090:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655025664:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757943.895122:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655025792:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757943.895144:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757943.895155:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757943/real 1787757943] req@ffff8c05a9f79500 x1874588655025792/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757943.895168:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655025792/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757943.895179:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757943.895183:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757943.895187:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655025792:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757943.895194:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655025920:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757943.895213:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757943.895223:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757943/real 1787757943] req@ffff8c05a9f7b100 x1874588655025920/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757943.895237:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655025920/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757943.895246:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757943.895250:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757943.895254:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655025920:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757943.895261:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655026048:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757943.895280:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757943.895294:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757943/real 1787757943] req@ffff8c05a9f79180 x1874588655026048/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757943.895318:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655026048/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757943.895333:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757943.895339:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757943.895347:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655026048:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757943.895360:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655026176:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757943.895400:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757943.895419:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757943/real 1787757943] req@ffff8c05a9f79f80 x1874588655026176/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757943.895436:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655026176/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757943.895446:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757943.895451:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757943.895455:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655026176:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757943.895462:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655026304:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757943.895485:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757943.895497:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757943/real 1787757943] req@ffff8c05a9f7bb80 x1874588655026304/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757943.895512:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655026304/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757943.895521:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757943.895525:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757943.895528:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655026304:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757944.021975:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757944.470189:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757945.046027:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757945.494085:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757946.070045:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757946.518218:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757947.094013:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757947.542220:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757948.118105:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757948.566219:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757949.013984:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757949.014001:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757949.014008:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757949.014023:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11005 00000100:00080000:3.0:1787757949.014040:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757949.014053:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757949.014108:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655026432/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757949.014146:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757949.014151:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757949.014153:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757949.014159:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11005 00000100:00080000:3.0:1787757949.014164:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757949.014169:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757949.014181:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757949.014196:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655026560/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757949.014209:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757949.014213:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757949.014216:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757949.014220:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11005 00000100:00080000:3.0:1787757949.014224:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757949.014230:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757949.014236:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757949.014249:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655026688/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757949.014264:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787757949.014269:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655026432:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757949.014270:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757949.014278:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757949.014292:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11005 00000100:00080000:3.0:1787757949.014297:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757949.014301:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757949.014308:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757949.014323:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655026816/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757949.014335:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757949.014340:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757949.014343:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757949.014347:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11005 00000100:00080000:3.0:1787757949.014351:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757949.014356:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757949.014362:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787757949.014373:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757949.014381:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655026944/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757949.014395:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757949.014399:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757949.014402:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757949.014407:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11005 00000100:00080000:3.0:1787757949.014413:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757949.014418:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757949.014424:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757949.014438:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655027072/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757949.014456:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757949/real 1787757949] req@ffff8c05a9f7bb80 x1874588655026432/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757949.014466:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757949.014480:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757949.014487:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757949.014499:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11005 00000100:00080000:3.0:1787757949.014506:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787757949.014509:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655026432/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757949.014521:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:1.0:1787757949.014532:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787757949.014537:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787757949.014542:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757949.014554:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655026432:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757949.014595:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655027200/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757949.014598:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655026560:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757949.014631:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757949.014636:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757949.014640:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757949.014646:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11005 00000100:00080000:3.0:1787757949.014654:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757949.014664:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:1.0:1787757949.014668:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:3.0:1787757949.014675:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757949.014697:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655027328/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757949.014710:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757949/real 1787757949] req@ffff8c05a9f79f80 x1874588655026560/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757949.014756:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655026560/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757949.014768:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757949.014772:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757949.014776:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655026560:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757949.014809:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655026688:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757949.014838:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757949.014853:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757949/real 1787757949] req@ffff8c05a9f79180 x1874588655026688/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757949.014867:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655026688/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757949.014922:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757949.014928:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757949.014932:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655026688:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757949.014940:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655026816:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757949.014966:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757949.014980:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757949/real 1787757949] req@ffff8c05a9f7b100 x1874588655026816/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757949.014994:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655026816/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757949.015013:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757949.015017:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757949.015021:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655026816:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757949.015027:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655026944:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757949.015046:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757949.015058:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757949/real 1787757949] req@ffff8c05a9f79500 x1874588655026944/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757949.015071:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655026944/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757949.015081:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757949.015084:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757949.015088:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655026944:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757949.015094:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655027072:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757949.015111:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757949.015122:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757949/real 1787757949] req@ffff8c05a9f78e00 x1874588655027072/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757949.015136:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655027072/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757949.015145:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757949.015149:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757949.015153:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655027072:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757949.015159:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655027200:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757949.015177:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757949.015189:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757949/real 1787757949] req@ffff8c05a9f7b800 x1874588655027200/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757949.015202:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655027200/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757949.015211:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757949.015214:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757949.015218:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655027200:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757949.015224:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655027328:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757949.015242:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757949.015253:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757949/real 1787757949] req@ffff8c05a9f7ad80 x1874588655027328/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757949.015266:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655027328/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757949.015276:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757949.015279:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757949.015283:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655027328:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757949.141986:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757949.590102:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757950.165994:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757950.614191:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757951.190061:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757951.638115:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757952.213996:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757952.662239:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757953.237993:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757953.686145:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757954.134127:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757954.134139:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787757954.134145:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757954.134159:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11010 00000100:00080000:0.0:1787757954.134172:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757954.134183:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757954.134233:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655027456/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757954.134272:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757954.134277:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757954.134280:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757954.134288:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11010 00000100:00080000:0.0:1787757954.134293:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757954.134298:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757954.134311:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757954.134334:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588655027584/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757954.134347:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655027456:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757954.134358:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757954.134368:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757954.134371:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757954.134378:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11010 00000100:00080000:0.0:1787757954.134385:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757954.134393:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757954.134402:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757954.134422:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588655027712/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757954.134438:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757954.134443:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757954.134446:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757954.134455:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11010 00000800:00000100:3.0:1787757954.134457:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757954.134465:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757954.134473:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757954.134521:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757954.134544:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655027840/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757954.134551:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757954/real 1787757954] req@ffff8c05a9f76300 x1874588655027456/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757954.134566:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757954.134577:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757954.134580:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757954.134586:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11010 00000100:00080000:0.0:1787757954.134591:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757954.134596:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757954.134602:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757954.134613:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655027968/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757954.134624:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757954.134628:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757954.134631:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757954.134635:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11010 00000100:00080000:0.0:1787757954.134638:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757954.134642:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757954.134646:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757954.134656:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588655028096/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757954.134666:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757954.134669:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787757954.134671:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757954.134676:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11010 00000100:00080000:0.0:1787757954.134680:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757954.134684:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757954.134689:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757954.134709:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655028224/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757954.134722:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655027456/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757954.134728:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757954.134741:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757954.134744:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757954.134750:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11010 00000100:00080000:0.0:1787757954.134754:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757954.134758:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757954.134763:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757954.134778:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655028352/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757954.134967:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757954.134974:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757954.134981:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655027456:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787757954.135021:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655027584:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757954.135124:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757954.135149:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757954/real 1787757954] req@ffff8c05a9f76a00 x1874588655027584/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757954.135167:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588655027584/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757954.135179:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757954.135183:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757954.135187:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655027584:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757954.135196:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655027712:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757954.135216:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757954.135228:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757954/real 1787757954] req@ffff8c05a9f77b80 x1874588655027712/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757954.135251:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588655027712/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757954.135261:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757954.135264:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757954.135268:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655027712:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757954.135275:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655027840:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757954.135294:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757954.135306:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757954/real 1787757954] req@ffff8c05a9f76680 x1874588655027840/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757954.135320:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655027840/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757954.135330:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757954.135333:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757954.135337:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655027840:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757954.135343:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655027968:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757954.135361:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757954.135372:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757954/real 1787757954] req@ffff8c05a9f77800 x1874588655027968/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757954.135386:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655027968/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757954.135403:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757954.135407:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757954.135411:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655027968:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757954.135417:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655028096:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757954.135436:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757954.135447:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757954/real 1787757954] req@ffff8c05a9f75c00 x1874588655028096/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757954.135460:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588655028096/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757954.135470:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757954.135474:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757954.135507:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655028096:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757954.135516:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655028224:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757954.135540:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757954.135554:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757954/real 1787757954] req@ffff8c05a9f74a80 x1874588655028224/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757954.135568:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655028224/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757954.135578:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757954.135582:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757954.135587:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655028224:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757954.135594:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655028352:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757954.135613:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757954.135624:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757954/real 1787757954] req@ffff8c05a9f74380 x1874588655028352/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757954.135638:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655028352/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757954.135648:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757954.135651:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757954.135654:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655028352:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757954.261995:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757954.710214:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757955.286008:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757955.734143:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757956.310066:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757956.758335:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757957.334027:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757957.782208:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757958.357977:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757958.806161:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757959.254107:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757959.254119:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787757959.254124:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757959.254147:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11015 00000100:00080000:1.0:1787757959.254162:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757959.254174:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757959.254231:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655028480/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757959.254295:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757959.254302:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757959.254304:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757959.254311:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11015 00000100:00080000:1.0:1787757959.254317:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757959.254323:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757959.254336:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757959.254366:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588655028608/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757959.254368:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655028480:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757959.254423:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757959.254446:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757959.254449:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757959.254455:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11015 00000100:00080000:1.0:1787757959.254462:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757959.254468:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757959.254477:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757959.254496:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655028736/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787757959.254510:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757959.254524:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757959.254530:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757959.254533:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757959.254539:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11015 00000100:00080000:1.0:1787757959.254544:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757959.254549:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757959.254556:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757959.254576:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757959/real 1787757959] req@ffff8c05b2d98380 x1874588655028480/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757959.254577:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655028864/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757959.254642:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757959.254647:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757959.254650:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757959.254655:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655028480/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757959.254664:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11015 00000100:00080000:1.0:1787757959.254677:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757959.254688:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:3.0:1787757959.254689:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787757959.254696:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787757959.254701:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757959.254712:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655028480:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787757959.254718:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655028992/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757959.254733:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757959.254738:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757959.254740:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757959.254747:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11015 00000100:00080000:1.0:1787757959.254754:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787757959.254755:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655028608:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787757959.254768:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757959.254778:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757959.254796:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655029120/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787757959.254811:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757959.254813:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757959.254820:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787757959.254824:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757959.254833:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11015 00000100:00080000:1.0:1787757959.254839:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757959.254849:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757959.254852:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757959/real 1787757959] req@ffff8c05b2d9a300 x1874588655028608/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787757959.254856:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757959.254933:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655029248/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757959.254953:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757959.254969:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757959.254974:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757959.254989:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11015 00000100:00080000:1.0:1787757959.254999:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787757959.255004:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588655028608/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757959.255014:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:3.0:1787757959.255032:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787757959.255036:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787757959.255043:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757959.255061:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655028608:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757959.255075:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655028736:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757959.255086:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655029376/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787757959.255122:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757959.255143:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757959/real 1787757959] req@ffff8c05b2d9aa00 x1874588655028736/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757959.255160:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655028736/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757959.255173:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757959.255178:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757959.255183:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655028736:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757959.255190:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655028864:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757959.255210:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757959.255222:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757959/real 1787757959] req@ffff8c05b2d9bb80 x1874588655028864/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757959.255255:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655028864/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757959.255265:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757959.255269:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757959.255273:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655028864:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757959.255279:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655028992:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757959.255297:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757959.255320:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757959/real 1787757959] req@ffff8c05b2d98700 x1874588655028992/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757959.255334:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655028992/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757959.255343:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757959.255346:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757959.255350:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655028992:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757959.255374:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655029120:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757959.255440:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757959.255465:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757959/real 1787757959] req@ffff8c05b2d9b100 x1874588655029120/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757959.255480:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655029120/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757959.255498:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757959.255502:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757959.255506:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655029120:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757959.255513:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655029248:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757959.255532:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757959.255543:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757959/real 1787757959] req@ffff8c05b2d99880 x1874588655029248/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757959.255557:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655029248/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757959.255566:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757959.255569:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757959.255573:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655029248:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757959.255579:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655029376:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757959.255596:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757959.255607:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757959/real 1787757959] req@ffff8c05b2d99f80 x1874588655029376/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757959.255620:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655029376/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757959.255629:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757959.255632:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757959.255636:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655029376:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757959.381988:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757959.830139:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757960.405991:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757960.854239:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757961.430032:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757961.878101:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757962.453999:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757962.902248:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757963.478017:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757963.926096:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757964.374017:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757964.374030:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757964.374035:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757964.374047:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11020 00000100:00080000:3.0:1787757964.374060:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757964.374069:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757964.374115:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655029504/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757964.374155:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757964.374160:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757964.374163:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757964.374169:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11020 00000100:00080000:3.0:1787757964.374173:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757964.374178:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757964.374190:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757964.374208:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655029632/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757964.374221:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655029504:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757964.374228:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757964.374236:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757964.374239:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757964.374246:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11020 00000100:00080000:3.0:1787757964.374250:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757964.374255:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757964.374261:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757964.374278:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655029760/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757964.374293:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757964.374345:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757964.374349:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757964.374364:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11020 00000800:00000100:1.0:1787757964.374367:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757964.374376:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757964.374387:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757964.374399:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757964.374424:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655029888/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757964.374440:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757964.374445:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757964.374448:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787757964.374456:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757964/real 1787757964] req@ffff8c05a9f7b480 x1874588655029504/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757964.374458:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11020 00000100:00080000:3.0:1787757964.374472:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757964.374484:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757964.374494:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757964.374505:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655029504/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757964.374515:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655030016/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757964.374538:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757964.374542:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757964.374551:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787757964.374553:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757964.374561:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787757964.374566:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655029504:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757964.374573:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11020 00000100:00080000:3.0:1787757964.374582:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757964.374586:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757964.374592:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757964.374604:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655030144/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757964.374610:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655029632:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757964.374637:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757964.374642:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757964.374645:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757964.374650:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11020 00000100:00080000:3.0:1787757964.374655:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757964.374661:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757964.374668:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787757964.374669:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757964.374693:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655030272/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757964.374701:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757964/real 1787757964] req@ffff8c05a9f78380 x1874588655029632/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757964.374730:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655029632/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757964.374740:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757964.374752:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757964.374754:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757964.374756:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757964.374768:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787757964.374771:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11020 00000100:00100000:1.0:1787757964.374780:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655029632:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757964.374783:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757964.374797:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757964.374804:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655029760:192.168.201.107@tcp:38:kworker.0 00000008:00100000:3.0:1787757964.374809:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757964.374832:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655030400/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787757964.374846:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757964.374869:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757964/real 1787757964] req@ffff8c05a9f78a80 x1874588655029760/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757964.374925:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655029760/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757964.374936:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757964.374940:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757964.374944:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655029760:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757964.374952:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655029888:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757964.374978:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757964.374992:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757964/real 1787757964] req@ffff8c05a9f7a680 x1874588655029888/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757964.375023:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655029888/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757964.375034:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757964.375038:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757964.375042:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655029888:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757964.375049:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655030016:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757964.375071:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757964.375083:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757964/real 1787757964] req@ffff8c05a9f7aa00 x1874588655030016/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757964.375107:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655030016/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757964.375116:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757964.375119:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757964.375123:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655030016:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757964.375144:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655030144:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757964.375164:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757964.375176:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757964/real 1787757964] req@ffff8c05a9f7a300 x1874588655030144/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757964.375190:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655030144/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757964.375200:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757964.375203:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757964.375207:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655030144:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757964.375213:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655030272:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757964.375232:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757964.375243:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757964/real 1787757964] req@ffff8c05a9f79880 x1874588655030272/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757964.375257:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655030272/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757964.375274:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757964.375278:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757964.375282:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655030272:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757964.375289:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655030400:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757964.375345:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757964.375369:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757964/real 1787757964] req@ffff8c05a9f78700 x1874588655030400/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757964.375395:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655030400/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757964.375413:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757964.375420:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757964.375427:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655030400:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757964.501996:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757964.950229:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757965.526016:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757965.974114:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757966.549981:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757966.998200:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757967.574039:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757968.022140:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757968.598013:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757969.046189:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757969.494048:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757969.494063:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757969.494070:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757969.494086:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11025 00000100:00080000:3.0:1787757969.494104:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757969.494115:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757969.494178:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655030528/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757969.494255:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757969.494261:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757969.494264:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757969.494270:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11025 00000100:00080000:3.0:1787757969.494276:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757969.494282:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757969.494296:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757969.494326:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655030656/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757969.494352:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757969.494357:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757969.494363:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787757969.494366:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655030528:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757969.494371:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11025 00000100:00080000:3.0:1787757969.494386:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757969.494391:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757969.494398:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757969.494415:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655030784/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757969.494429:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757969.494434:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757969.494436:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757969.494441:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11025 00000100:00080000:3.0:1787757969.494446:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757969.494451:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757969.494457:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757969.494470:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655030912/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757969.494483:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757969.494487:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757969.494490:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787757969.494499:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787757969.494506:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11025 00000100:00080000:3.0:1787757969.494515:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757969.494522:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757969.494528:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757969.494544:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655031040/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757969.494556:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757969.494560:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757969.494564:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757969.494570:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11025 00000100:00080000:3.0:1787757969.494574:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787757969.494578:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757969/real 1787757969] req@ffff8c05a9f78700 x1874588655030528/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757969.494584:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757969.494596:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757969.494622:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655031168/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757969.494632:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655030528/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757969.494639:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757969.494652:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757969.494657:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757969.494672:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11025 00000100:00080000:3.0:1787757969.494681:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757969.494688:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757969.494692:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757969.494703:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655031296/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757969.494706:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757969.494730:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787757969.494739:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757969.494748:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787757969.494750:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655030528:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757969.494751:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757969.494766:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11025 00000100:00080000:3.0:1787757969.494770:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757969.494775:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757969.494781:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757969.494794:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655031424/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757969.494807:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655030656:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757969.494851:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757969.494870:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757969/real 1787757969] req@ffff8c05a9f79880 x1874588655030656/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757969.494944:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655030656/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757969.494956:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757969.494961:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757969.494966:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655030656:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757969.494974:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655030784:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757969.495000:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757969.495014:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757969/real 1787757969] req@ffff8c05a9f7a300 x1874588655030784/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757969.495029:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655030784/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757969.495039:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757969.495043:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757969.495048:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655030784:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757969.495054:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655030912:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757969.495076:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757969.495087:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757969/real 1787757969] req@ffff8c05a9f7aa00 x1874588655030912/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757969.495102:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655030912/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757969.495112:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757969.495115:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757969.495119:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655030912:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757969.495125:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655031040:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757969.495144:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757969.495155:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757969/real 1787757969] req@ffff8c05a9f7a680 x1874588655031040/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757969.495169:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655031040/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757969.495179:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757969.495182:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757969.495186:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655031040:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757969.495192:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655031168:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757969.495240:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757969.495257:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757969/real 1787757969] req@ffff8c05a9f78a80 x1874588655031168/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757969.495282:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655031168/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757969.495293:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757969.495296:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757969.495300:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655031168:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757969.495308:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655031296:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757969.495333:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757969.495344:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757969/real 1787757969] req@ffff8c05a9f78380 x1874588655031296/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757969.495359:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655031296/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757969.495369:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757969.495372:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757969.495376:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655031296:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757969.495402:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655031424:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757969.495421:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757969.495432:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757969/real 1787757969] req@ffff8c05a9f7b480 x1874588655031424/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757969.495447:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655031424/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757969.495465:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757969.495469:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757969.495473:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655031424:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757969.622047:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757970.070125:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757970.645986:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757971.094316:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757971.670025:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757972.118123:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757972.694044:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757973.142304:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757973.718028:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757974.166120:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787757974.614008:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757974.614026:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787757974.614032:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757974.614050:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11031 00000100:00080000:3.0:1787757974.614067:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757974.614080:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757974.614168:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655031552/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757974.614211:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757974.614216:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757974.614219:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757974.614227:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11031 00000100:00080000:3.0:1787757974.614232:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757974.614238:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757974.614252:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757974.614269:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655031680/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757974.614283:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757974.614288:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757974.614291:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757974.614296:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11031 00000100:00080000:3.0:1787757974.614301:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757974.614305:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757974.614311:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757974.614323:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655031808/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757974.614337:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787757974.614343:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655031552:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757974.614344:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787757974.614351:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757974.614378:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11031 00000100:00080000:3.0:1787757974.614382:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757974.614388:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757974.614397:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757974.614414:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655031936/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757974.614428:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757974.614433:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787757974.614435:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757974.614440:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11031 00000100:00080000:3.0:1787757974.614446:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757974.614451:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757974.614456:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787757974.614471:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757974.614475:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655032064/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757974.614490:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757974.614496:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787757974.614499:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757974.614504:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11031 00000100:00080000:3.0:1787757974.614509:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757974.614513:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757974.614518:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757974.614533:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655032192/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757974.614549:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757974/real 1787757974] req@ffff8c05a9f7ad80 x1874588655031552/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757974.614563:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757974.614577:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787757974.614582:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757974.614594:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11031 00000100:00080000:3.0:1787757974.614600:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787757974.614607:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655031552/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787757999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757974.614609:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787757974.614621:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787757974.614641:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787757974.614648:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655032320/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757974.614655:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757974.614672:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655031552:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787757974.614678:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757974.614686:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787757974.614692:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655031680:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787757974.614694:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757974.614712:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11031 00000100:00080000:3.0:1787757974.614720:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757974.614729:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:1.0:1787757974.614739:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:3.0:1787757974.614744:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757974.614758:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757974/real 1787757974] req@ffff8c05a9f7b800 x1874588655031680/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757974.614785:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655031680/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757974.614801:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655032448/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757974.614802:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757974.614820:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757974.614835:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655031680:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757974.614929:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655031808:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757974.614958:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757974.614973:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757974/real 1787757974] req@ffff8c05a9f78e00 x1874588655031808/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757974.614988:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655031808/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757974.614999:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757974.615003:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757974.615007:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655031808:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757974.615014:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655031936:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757974.615035:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757974.615046:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757974/real 1787757974] req@ffff8c05a9f79500 x1874588655031936/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757974.615060:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655031936/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757974.615071:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757974.615074:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757974.615077:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655031936:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757974.615084:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655032064:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757974.615104:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757974.615115:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757974/real 1787757974] req@ffff8c05a9f7b100 x1874588655032064/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757974.615167:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655032064/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757974.615178:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757974.615183:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757974.615187:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655032064:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757974.615193:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655032192:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757974.615218:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757974.615231:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757974/real 1787757974] req@ffff8c05a9f79180 x1874588655032192/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757974.615245:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655032192/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787757999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757974.615255:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757974.615259:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757974.615262:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655032192:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757974.615269:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655032320:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757974.615290:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757974.615301:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757974/real 1787757974] req@ffff8c05a9f79f80 x1874588655032320/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757974.615324:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655032320/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757974.615334:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757974.615338:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757974.615341:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655032320:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757974.615348:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655032448:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757974.615367:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757974.615378:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757974/real 1787757974] req@ffff8c05a9f7bb80 x1874588655032448/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757974.615392:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655032448/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787757999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757974.615402:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757974.615406:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757974.615409:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655032448:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757974.742022:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757975.190273:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757975.766012:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757976.214131:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757976.789980:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757977.238266:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757977.814002:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757978.262169:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757978.838041:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757979.286325:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757979.734124:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757979.734146:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787757979.734154:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757979.734175:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11036 00000100:00080000:0.0:1787757979.734198:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757979.734213:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757979.734281:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655032576/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757979.734327:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757979.734332:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757979.734335:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757979.734342:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11036 00000100:00080000:0.0:1787757979.734348:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757979.734353:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757979.734366:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757979.734386:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655032704/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757979.734401:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757979.734408:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757979.734414:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787757979.734418:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655032576:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757979.734424:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11036 00000100:00080000:0.0:1787757979.734437:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757979.734443:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757979.734450:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757979.734469:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588655032832/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757979.734491:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757979.734499:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757979.734503:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757979.734511:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11036 00000100:00080000:0.0:1787757979.734536:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787757979.734545:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757979.734554:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757979.734569:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757979.734591:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655032960/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757979.734606:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757979.734611:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757979.734615:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757979.734623:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11036 00000100:00100000:1.0:1787757979.734624:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757979/real 1787757979] req@ffff8c05a9f74380 x1874588655032576/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757979.734631:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757979.734644:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757979.734656:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757979.734667:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655032576/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757979.734686:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787757979.734689:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655033088/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757979.734698:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757979.734715:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655032576:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757979.734721:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757979.734729:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757979.734732:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757979.734740:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11036 00000100:00080000:0.0:1787757979.734745:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757979.734749:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757979.734756:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757979.734763:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655032704:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787757979.734775:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588655033216/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757979.734794:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757979.734799:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1787757979.734807:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757979.734808:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757979.734823:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11036 00000100:00080000:0.0:1787757979.734832:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787757979.734834:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757979/real 1787757979] req@ffff8c05a9f74a80 x1874588655032704/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757979.734848:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757979.734866:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757979.734866:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655032704/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757979.734950:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787757979.734952:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588655033344/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757979.734963:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757979.734977:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655032704:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757979.734990:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787757979.734999:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655032832:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757979.735001:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757979.735009:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757979.735024:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11036 00000100:00080000:0.0:1787757979.735057:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757979.735068:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757979.735080:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787757979.735094:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787757979.735111:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655033472/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787757979.735129:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757979/real 1787757979] req@ffff8c05a9f75c00 x1874588655032832/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757979.735157:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588655032832/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757979.735169:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757979.735175:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757979.735180:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655032832:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757979.735190:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655032960:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757979.735220:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757979.735234:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757979/real 1787757979] req@ffff8c05a9f77800 x1874588655032960/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757979.735249:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655032960/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757979.735259:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757979.735262:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757979.735266:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655032960:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757979.735273:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655033088:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757979.735293:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757979.735305:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757979/real 1787757979] req@ffff8c05a9f76680 x1874588655033088/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757979.735319:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655033088/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757979.735329:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757979.735333:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757979.735336:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655033088:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757979.735359:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655033216:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787757979.735379:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757979.735391:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757979/real 1787757979] req@ffff8c05a9f77b80 x1874588655033216/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757979.735406:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588655033216/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757979.735417:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757979.735420:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757979.735423:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655033216:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757979.735430:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655033344:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757979.735449:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757979.735460:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757979/real 1787757979] req@ffff8c05a9f76a00 x1874588655033344/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757979.735475:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588655033344/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757979.735484:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757979.735488:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757979.735491:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655033344:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787757979.735497:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655033472:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787757979.735515:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757979.735526:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757979/real 1787757979] req@ffff8c05a9f76300 x1874588655033472/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757979.735539:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655033472/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757979.735558:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757979.735562:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787757979.735566:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655033472:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757979.861979:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757980.310169:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757980.886042:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757981.334289:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757981.909960:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757982.358045:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757982.933970:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757983.382218:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757983.958062:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757984.406124:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757984.854082:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757984.854098:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787757984.854106:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757984.854122:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11041 00000100:00080000:1.0:1787757984.854141:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757984.854153:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757984.854218:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655033600/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757984.854268:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757984.854273:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757984.854276:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757984.854281:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11041 00000100:00080000:1.0:1787757984.854286:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757984.854290:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757984.854304:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757984.854320:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655033728/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757984.854336:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757984.854341:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757984.854344:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757984.854348:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11041 00000100:00080000:1.0:1787757984.854352:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757984.854356:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757984.854362:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757984.854376:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655033856/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757984.854395:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787757984.854398:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655033600:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787757984.854402:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757984.854411:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757984.854430:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11041 00000100:00080000:1.0:1787757984.854435:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757984.854439:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757984.854447:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757984.854464:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655033984/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757984.854479:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757984.854484:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757984.854487:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757984.854492:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11041 00000100:00080000:1.0:1787757984.854499:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757984.854508:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757984.854516:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787757984.854528:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787757984.854543:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655034112/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757984.854559:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757984.854563:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757984.854566:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757984.854572:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11041 00000100:00080000:1.0:1787757984.854579:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757984.854587:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757984.854599:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757984.854606:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757984/real 1787757984] req@ffff8c05b2d98000 x1874588655033600/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757984.854647:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655034240/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757984.854655:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655033600/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757984.854706:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757984.854735:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787757984.854746:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757984.854768:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11041 00000100:00080000:3.0:1787757984.854771:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787757984.854780:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787757984.854790:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787757984.854804:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757984.854811:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655033600:192.168.201.107@tcp:250:kworker.0 00000008:00100000:1.0:1787757984.854818:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757984.854850:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655033728:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787757984.854857:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655034368/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757984.854983:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787757984.854993:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757984.855001:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757984.855007:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757984.855016:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11041 00000100:00080000:1.0:1787757984.855025:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787757984.855033:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757984/real 1787757984] req@ffff8c05b2d9b480 x1874588655033728/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757984.855035:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757984.855061:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757984.855083:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655033728/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757984.855094:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655034496/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787757984.855126:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757984.855152:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757984.855161:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655033728:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757984.855212:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655033856:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757984.855243:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757984.855257:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757984/real 1787757984] req@ffff8c05b2d99f80 x1874588655033856/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757984.855271:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655033856/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757984.855281:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757984.855285:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757984.855289:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655033856:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757984.855298:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655033984:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757984.855319:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757984.855331:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757984/real 1787757984] req@ffff8c05b2d99880 x1874588655033984/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757984.855345:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655033984/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757984.855355:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757984.855358:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757984.855362:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655033984:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757984.855369:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655034112:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757984.855388:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757984.855400:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757984/real 1787757984] req@ffff8c05b2d9b100 x1874588655034112/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757984.855413:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655034112/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757984.855423:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757984.855426:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757984.855430:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655034112:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757984.855437:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655034240:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757984.855456:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757984.855466:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757984/real 1787757984] req@ffff8c05b2d98700 x1874588655034240/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757984.855481:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655034240/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757984.855491:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757984.855494:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757984.855498:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655034240:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757984.855504:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655034368:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757984.855525:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757984.855536:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757984/real 1787757984] req@ffff8c05b2d9bb80 x1874588655034368/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757984.855549:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655034368/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757984.855559:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757984.855562:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757984.855566:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655034368:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757984.855572:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655034496:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757984.855591:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757984.855602:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757984/real 1787757984] req@ffff8c05b2d9aa00 x1874588655034496/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757984.855615:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655034496/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757984.855634:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757984.855638:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757984.855641:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655034496:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757984.982035:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757985.430273:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757986.006038:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757986.454187:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757987.030053:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757987.478269:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757988.054073:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757988.502162:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757989.078031:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757989.526239:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787757989.974109:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757989.974122:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787757989.974128:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757989.974144:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11046 00000100:00080000:0.0:1787757989.974159:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757989.974171:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787757989.974224:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588655034624/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757989.974259:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757989.974264:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757989.974266:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757989.974271:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11046 00000100:00080000:0.0:1787757989.974275:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757989.974279:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757989.974290:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757989.974305:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588655034752/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757989.974319:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757989.974323:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757989.974326:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757989.974332:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11046 00000100:00080000:0.0:1787757989.974336:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757989.974341:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757989.974349:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757989.974361:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588655034880/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757989.974376:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757989.974380:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757989.974383:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757989.974390:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11046 00000100:00080000:0.0:1787757989.974400:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787757989.974403:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655034624:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757989.974409:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757989.974427:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757989.974444:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588655035008/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787757989.974459:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757989.974465:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787757989.974470:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757989.974482:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11046 00000100:00080000:0.0:1787757989.974492:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757989.974501:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757989.974511:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787757989.974538:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655035136/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787757989.974545:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757989.974573:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757989.974581:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787757989.974584:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787757989.974593:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11046 00000100:00080000:0.0:1787757989.974604:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757989.974612:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757989.974624:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757989.974635:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757989/real 1787757989] req@ffff8c05a9f76d80 x1874588655034624/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787757989.974659:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588655035264/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757989.974673:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588655034624/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757989.974718:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787757989.974723:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787757989.974726:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787757989.974733:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787757989.974735:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787757989.974752:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11046 00000100:00100000:3.0:1787757989.974754:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655034624:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787757989.974767:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787757989.974779:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655034752:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787757989.974781:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787757989.974811:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787757989.974927:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588655035392/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787757989.974927:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787757989.974965:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787757989.974971:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787757989.974979:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787757989.974980:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757989/real 1787757989] req@ffff8c05a9f77100 x1874588655034752/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787757989.975001:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11046 00000100:00080000:0.0:1787757989.975021:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787757989.975039:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787757989.975047:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588655034752/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787757989.975053:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787757989.975079:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757989.975089:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787757989.975096:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655035520/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757989.975107:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655034752:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757989.975186:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655034880:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757989.975226:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757989.975242:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757989/real 1787757989] req@ffff8c05a9f75180 x1874588655034880/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757989.975265:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588655034880/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757989.975277:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757989.975281:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757989.975286:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655034880:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757989.975306:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655035008:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757989.975336:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757989.975350:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757989/real 1787757989] req@ffff8c05a9f74000 x1874588655035008/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757989.975365:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588655035008/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757989.975376:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757989.975380:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757989.975384:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655035008:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757989.975392:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655035136:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757989.975425:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757989.975466:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757989/real 1787757989] req@ffff8c05a9f75880 x1874588655035136/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757989.975494:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655035136/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757989.975505:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757989.975509:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757989.975513:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655035136:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757989.975521:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655035264:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757989.975547:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757989.975564:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757989/real 1787757989] req@ffff8c05a9f75500 x1874588655035264/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757989.975580:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588655035264/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757989.975590:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757989.975594:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757989.975599:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655035264:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757989.975606:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655035392:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757989.976118:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757989.976145:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757989/real 1787757989] req@ffff8c05a9f74e00 x1874588655035392/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757989.976185:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588655035392/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757989.976198:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757989.976204:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757989.976211:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655035392:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757989.976220:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655035520:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757989.976248:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757989.976262:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757989/real 1787757989] req@ffff8c05a9f76300 x1874588655035520/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757989.976276:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655035520/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757989.976286:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757989.976290:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757989.976293:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655035520:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757990.102052:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757990.550130:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757991.126006:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757991.574257:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757992.150018:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757992.598114:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757993.174033:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787757993.622251:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757994.198032:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757994.646107:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787757995.094058:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757995.094075:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787757995.094083:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757995.094099:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11051 00000100:00080000:1.0:1787757995.094115:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757995.094127:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787757995.094192:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588655035648/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757995.094241:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757995.094246:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757995.094250:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757995.094256:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11051 00000100:00080000:1.0:1787757995.094261:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757995.094266:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757995.094281:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757995.094298:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655035776/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757995.094313:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757995.094318:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787757995.094321:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757995.094326:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11051 00000100:00080000:1.0:1787757995.094330:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757995.094335:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757995.094340:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787757995.094357:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655035648:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787757995.094359:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655035904/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757995.094386:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757995.094390:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757995.094392:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757995.094396:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11051 00000100:00080000:1.0:1787757995.094400:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757995.094404:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757995.094410:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787757995.094428:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655036032/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757995.094442:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757995.094447:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787757995.094449:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757995.094453:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11051 00000100:00080000:1.0:1787757995.094459:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757995.094464:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757995.094469:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757995.094483:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655036160/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757995.094494:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757995.094503:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:3.0:1787757995.094504:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757995.094508:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757995.094518:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11051 00000100:00080000:1.0:1787757995.094523:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757995.094528:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757995.094536:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787757995.094562:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655036288/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787757995.094581:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787757995.094583:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757995/real 1787757995] req@ffff8c05b2d9a300 x1874588655035648/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757995.094597:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787757995.094607:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757995.094623:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11051 00000100:00100000:3.0:1787757995.094630:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588655035648/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787757995.094636:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757995.094654:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:3.0:1787757995.094658:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787757995.094664:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787757995.094669:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757995.094679:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655035648:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787757995.094693:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655036416/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757995.094696:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655035776:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787757995.094722:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787757995.094728:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787757995.094732:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787757995.094740:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11051 00000800:00000100:3.0:1787757995.094748:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787757995.094750:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787757995.094785:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787757995.094795:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787757995.094809:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757995/real 1787757995] req@ffff8c05b2d98380 x1874588655035776/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787757995.094819:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655036544/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787757995.094937:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655035776/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757995.094960:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757995.094968:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757995.094976:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655035776:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757995.095019:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655035904:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757995.095054:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757995.095071:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757995/real 1787757995] req@ffff8c05b2d9aa00 x1874588655035904/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757995.095087:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655035904/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757995.095099:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757995.095104:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757995.095108:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655035904:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757995.095117:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655036032:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757995.095139:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757995.095151:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757995/real 1787757995] req@ffff8c05b2d9bb80 x1874588655036032/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757995.095166:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655036032/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757995.095176:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757995.095180:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757995.095183:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655036032:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757995.095190:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655036160:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757995.095211:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757995.095222:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757995/real 1787757995] req@ffff8c05b2d98700 x1874588655036160/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757995.095236:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655036160/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757995.095246:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757995.095249:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757995.095253:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655036160:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757995.095259:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655036288:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787757995.095278:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757995.095289:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757995/real 1787757995] req@ffff8c05b2d9b100 x1874588655036288/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757995.095304:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655036288/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757995.095314:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757995.095318:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757995.095321:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655036288:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787757995.095327:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655036416:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757995.095345:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757995.095356:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757995/real 1787757995] req@ffff8c05b2d99880 x1874588655036416/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757995.095369:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655036416/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757995.095379:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757995.095383:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757995.095387:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655036416:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787757995.095393:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655036544:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787757995.095411:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787757995.095421:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787757995/real 1787757995] req@ffff8c05b2d99f80 x1874588655036544/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787757995.095435:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655036544/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787757995.095454:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787757995.095458:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787757995.095462:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655036544:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787757995.222015:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757995.670328:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757996.246005:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757996.694092:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757997.270022:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757997.718212:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757998.294000:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787757998.742164:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787757999.317964:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787757999.766076:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758000.214012:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758000.214024:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758000.214029:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758000.214042:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11056 00000100:00080000:0.0:1787758000.214056:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758000.214065:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758000.214113:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588655036672/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758000.214146:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758000.214149:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758000.214151:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758000.214156:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11056 00000100:00080000:0.0:1787758000.214159:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758000.214163:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758000.214191:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758000.214209:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588655036800/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758000.214222:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758000.214228:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758000.214230:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758000.214236:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11056 00000100:00080000:0.0:1787758000.214242:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758000.214247:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758000.214253:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758000.214268:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655036928/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787758000.214284:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655036672:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758000.214287:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758000.214317:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758000.214322:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758000.214333:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11056 00000100:00080000:0.0:1787758000.214341:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758000.214350:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758000.214361:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758000.214389:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655037056/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758000.214411:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758000.214419:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758000.214423:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758000.214431:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11056 00000100:00080000:0.0:1787758000.214438:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758000.214446:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758000.214454:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758000.214467:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655036800:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758000.214480:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588655037184/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758000.214510:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758000.214514:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758000.214517:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758000.214521:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11056 00000100:00080000:0.0:1787758000.214527:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758000.214533:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758000.214539:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758000.214550:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655036928:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758000.214576:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655037312/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758000.214603:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787758000.214607:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655037056:192.168.201.107@tcp:8:kworker.0 00000100:00080000:0.0:1787758000.214619:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758000.214627:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758000.214640:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11056 00000100:00100000:3.0:1787758000.214648:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655037184:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758000.214650:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758000.214711:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758000.214721:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758000.214739:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655037440/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758000.214751:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787758000.214752:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655037312:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758000.214759:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758000.214764:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758000.214769:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11056 00000100:00080000:0.0:1787758000.214775:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758000.214780:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758000.214786:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758000.214788:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655037440:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758000.214803:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655037568/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787758000.214834:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655037568:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758000.342080:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758000.790107:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758001.365950:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758001.814145:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758002.389961:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758002.838076:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787758003.286301:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.107:988 00000400:00000100:3.0:1787758003.286347:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.107@tcp at host 192.168.201.107 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787758003.286361:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:3.0:1787758003.286403:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:3.0:1787758003.286416:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:3.0:1787758003.286440:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:3.0:1787758003.286456:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:1.0:1787758003.286457:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758000/real 1787758003] req@ffff8c05a9f76680 x1874588655036928/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787758003.286477:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:3.0:1787758003.286500:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:1.0:1787758003.286505:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655036928/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787758003.286519:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00080000:1.0:1787758003.286525:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758003.286536:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758003.286543:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655036928:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758003.286555:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758000/real 1787758003] req@ffff8c05a9f77800 x1874588655037056/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758003.286573:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655037056/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758003.286583:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758003.286587:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758003.286590:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655037056:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758003.286599:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758000/real 1787758003] req@ffff8c05a9f75c00 x1874588655037184/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758003.286612:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588655037184/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758003.286637:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758003.286641:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758003.286645:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655037184:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758003.286653:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758000/real 1787758003] req@ffff8c05a9f76a00 x1874588655036672/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758003.286667:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588655036672/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758003.286675:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758003.286680:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758003.286683:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655036672:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758003.286691:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758000/real 1787758003] req@ffff8c05a9f77b80 x1874588655036800/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758003.286704:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588655036800/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758003.286712:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758003.286716:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758003.286719:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655036800:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758003.286768:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758000/real 1787758003] req@ffff8c05a9f76300 x1874588655037568/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758003.286784:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655037568/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758003.286793:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758003.286797:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758003.286801:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655037568:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758003.286810:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758000/real 1787758003] req@ffff8c05a9f74380 x1874588655037440/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758003.286825:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655037440/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758003.286835:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758003.286840:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758003.286844:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655037440:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758003.286853:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758000/real 1787758003] req@ffff8c05a9f74a80 x1874588655037312/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758003.286866:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655037312/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758003.286929:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758003.286935:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758003.286940:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655037312:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758003.413956:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758003.862112:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758004.438025:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758004.886068:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758005.334012:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758005.334042:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758005.334049:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758005.334064:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11061 00000100:00080000:0.0:1787758005.334081:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758005.334094:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758005.334155:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588655037696/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758005.334198:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758005.334204:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758005.334206:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758005.334213:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11061 00000100:00080000:0.0:1787758005.334217:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758005.334223:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758005.334237:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758005.334254:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588655037824/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758005.334270:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758005.334279:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758005.334282:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758005.334283:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655037696:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758005.334291:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11061 00000100:00080000:0.0:1787758005.334299:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758005.334307:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758005.334313:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758005.334328:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655037952/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758005.334345:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758005.334352:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758005.334357:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758005.334364:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11061 00000100:00080000:0.0:1787758005.334402:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758005.334412:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:1.0:1787758005.334425:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:0.0:1787758005.334427:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758005.334460:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588655038080/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758005.334491:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758005.334495:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758005.334497:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758005.334502:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11061 00000100:00080000:0.0:1787758005.334507:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758005.334514:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758005.334514:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758005/real 1787758005] req@ffff8c05a9f74e00 x1874588655037696/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787758005.334524:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758005.334556:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588655038208/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758005.334569:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758005.334572:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588655037696/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758005.334574:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758005.334600:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758005.334611:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11061 00000100:00080000:0.0:1787758005.334621:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758005.334622:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758005.334629:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:1.0:1787758005.334632:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000008:00100000:0.0:1787758005.334639:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758005.334642:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655037696:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787758005.334660:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588655038336/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758005.334673:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758005.334677:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758005.334679:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758005.334684:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655037824:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758005.334689:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11061 00000100:00080000:0.0:1787758005.334697:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758005.334703:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758005.334710:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787758005.334728:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758005.334730:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588655038464/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758005.334748:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758005.334753:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758005/real 1787758005] req@ffff8c05a9f75500 x1874588655037824/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758005.334755:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758005.334760:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758005.334771:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11061 00000100:00080000:0.0:1787758005.334780:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758005.334786:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588655037824/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758005.334789:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758005.334800:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787758005.334806:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758005.334814:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758005.334820:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588655038592/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758005.334823:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655037824:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758005.334840:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655037952:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758005.334902:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758005.334919:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758005/real 1787758005] req@ffff8c05a9f75880 x1874588655037952/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758005.334933:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655037952/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758005.334942:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758005.334946:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758005.334950:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655037952:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758005.334957:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655038080:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758005.334982:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758005.334993:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758005/real 1787758005] req@ffff8c05a9f74000 x1874588655038080/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758005.335007:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588655038080/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758005.335017:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758005.335020:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758005.335024:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655038080:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758005.335030:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655038208:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758005.335052:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758005.335065:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758005/real 1787758005] req@ffff8c05a9f75180 x1874588655038208/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758005.335086:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588655038208/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758005.335095:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758005.335099:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758005.335102:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655038208:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758005.335109:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655038336:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758005.335132:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758005.335145:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758005/real 1787758005] req@ffff8c05a9f77100 x1874588655038336/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758005.335159:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588655038336/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758005.335169:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758005.335173:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758005.335179:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655038336:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758005.335204:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655038464:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758005.335229:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758005.335244:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758005/real 1787758005] req@ffff8c05a9f76d80 x1874588655038464/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758005.335258:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588655038464/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758005.335268:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758005.335272:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758005.335276:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655038464:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758005.335284:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655038592:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758005.335309:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758005.335324:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758005/real 1787758005] req@ffff8c05a9f74e00 x1874588655038592/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758005.335339:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588655038592/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758005.335348:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758005.335352:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758005.335356:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655038592:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758005.461954:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758005.910121:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758006.485974:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758006.934085:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758007.509947:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758007.958085:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758008.533965:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758008.982150:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758009.558033:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758010.006234:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758010.454016:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758010.454033:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758010.454039:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758010.454056:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11066 00000100:00080000:0.0:1787758010.454074:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758010.454086:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758010.454153:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655038720/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758010.454204:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758010.454210:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758010.454213:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758010.454218:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11066 00000100:00080000:0.0:1787758010.454223:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758010.454228:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758010.454242:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758010.454259:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655038848/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758010.454275:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758010.454280:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758010.454282:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758010.454288:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11066 00000100:00080000:0.0:1787758010.454291:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758010.454296:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758010.454301:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758010.454319:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655038720:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787758010.454320:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655038976/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758010.454348:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758010.454352:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758010.454354:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758010.454359:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11066 00000100:00080000:0.0:1787758010.454365:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758010.454369:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758010.454377:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758010.454393:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588655039104/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758010.454407:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758010.454412:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758010.454415:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758010.454419:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11066 00000100:00080000:0.0:1787758010.454436:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758010.454442:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758010.454448:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787758010.454455:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758010.454471:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588655039232/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758010.454486:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758010.454522:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758010.454524:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758010.454530:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11066 00000100:00080000:0.0:1787758010.454536:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758010.454541:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758010.454549:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758010.454562:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758010/real 1787758010] req@ffff8c05a9f74a80 x1874588655038720/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758010.454569:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588655039360/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758010.454614:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758010.454619:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758010.454622:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758010.454630:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655038720/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758010.454635:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11066 00000100:00080000:0.0:1787758010.454647:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758010.454659:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758010.454669:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787758010.454672:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758010.454684:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758010.454686:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655039488/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758010.454695:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655038720:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758010.454717:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758010.454722:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655038848:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758010.454726:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758010.454734:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758010.454743:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11066 00000100:00080000:0.0:1787758010.454748:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758010.454752:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758010.454760:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787758010.454776:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758010.454781:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655039616/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758010.454808:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758010/real 1787758010] req@ffff8c05a9f74380 x1874588655038848/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758010.454839:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655038848/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758010.454935:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758010.454944:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758010.454952:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655038848:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758010.455004:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655038976:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758010.455051:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758010.455070:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758010/real 1787758010] req@ffff8c05a9f76300 x1874588655038976/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758010.455087:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655038976/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758010.455098:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758010.455102:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758010.455106:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655038976:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758010.455114:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655039104:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758010.455138:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758010.455150:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758010/real 1787758010] req@ffff8c05a9f77b80 x1874588655039104/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758010.455165:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588655039104/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758010.455175:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758010.455178:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758010.455182:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655039104:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758010.455189:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655039232:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758010.455208:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758010.455219:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758010/real 1787758010] req@ffff8c05a9f76a00 x1874588655039232/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758010.455233:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588655039232/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758010.455252:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758010.455256:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758010.455260:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655039232:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758010.455266:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655039360:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758010.455284:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758010.455295:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758010/real 1787758010] req@ffff8c05a9f75c00 x1874588655039360/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758010.455308:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588655039360/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758010.455319:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758010.455322:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758010.455326:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655039360:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758010.455332:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655039488:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758010.455350:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758010.455360:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758010/real 1787758010] req@ffff8c05a9f77800 x1874588655039488/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758010.455373:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655039488/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758010.455383:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758010.455387:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758010.455392:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655039488:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758010.455398:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655039616:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758010.455417:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758010.455428:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758010/real 1787758010] req@ffff8c05a9f76680 x1874588655039616/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758010.455442:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655039616/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758010.455451:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758010.455454:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758010.455458:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655039616:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758010.582016:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758011.030051:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758011.606035:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758012.054135:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758012.629983:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758013.078044:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758013.653949:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758014.102170:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758014.677985:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758015.126137:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758015.574047:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758015.574063:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758015.574069:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758015.574085:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11071 00000100:00080000:1.0:1787758015.574103:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758015.574114:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758015.574184:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655039744/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758015.574228:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758015.574233:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758015.574236:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758015.574243:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11071 00000100:00080000:1.0:1787758015.574248:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758015.574253:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758015.574267:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758015.574285:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655039872/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758015.574306:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758015.574314:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758015.574318:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758015.574327:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11071 00000100:00080000:1.0:1787758015.574335:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758015.574344:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758015.574355:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758015.574360:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655039744:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758015.574386:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655040000/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758015.574435:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758015.574441:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758015.574444:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758015.574451:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11071 00000100:00080000:1.0:1787758015.574457:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758015.574464:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758015.574473:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758015.574492:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655040128/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758015.574516:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787758015.574528:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758015.574529:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758015.574537:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758015.574546:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11071 00000100:00080000:1.0:1787758015.574553:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758015.574560:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758015.574567:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758015.574584:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655040256/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758015.574597:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758015.574602:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758015.574605:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758015.574613:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11071 00000100:00100000:3.0:1787758015.574614:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758015/real 1787758015] req@ffff8c05b2d9b480 x1874588655039744/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758015.574618:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758015.574630:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758015.574644:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758015.574663:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655040384/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787758015.574671:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655039744/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758015.574688:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758015.574706:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758015.574711:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758015.574714:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758015.574720:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11071 00000100:00080000:3.0:1787758015.574728:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787758015.574729:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787758015.574743:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655039744:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758015.574744:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758015.574757:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758015.574767:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655039872:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758015.574776:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655040512/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758015.574792:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787758015.574811:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758015.574812:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758015.574821:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758015.574830:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11071 00000100:00080000:1.0:1787758015.574840:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787758015.574850:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758015/real 1787758015] req@ffff8c05b2d98000 x1874588655039872/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758015.574856:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758015.574938:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758015.574969:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655040640/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787758015.574976:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655039872/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758015.575019:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758015.575028:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758015.575040:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655039872:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758015.575090:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655040000:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758015.575125:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758015.575141:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758015/real 1787758015] req@ffff8c05b2d99f80 x1874588655040000/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758015.575157:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655040000/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758015.575168:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758015.575173:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758015.575177:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655040000:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758015.575185:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655040128:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758015.575206:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758015.575218:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758015/real 1787758015] req@ffff8c05b2d99880 x1874588655040128/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758015.575233:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655040128/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758015.575243:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758015.575246:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758015.575250:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655040128:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758015.575256:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655040256:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758015.575275:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758015.575287:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758015/real 1787758015] req@ffff8c05b2d9b100 x1874588655040256/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758015.575301:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655040256/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758015.575311:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758015.575314:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758015.575318:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655040256:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758015.575324:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655040384:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758015.575342:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758015.575353:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758015/real 1787758015] req@ffff8c05b2d98700 x1874588655040384/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758015.575368:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655040384/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758015.575378:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758015.575381:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758015.575385:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655040384:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758015.575391:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655040512:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758015.575457:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758015.575473:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758015/real 1787758015] req@ffff8c05b2d9bb80 x1874588655040512/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758015.575487:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655040512/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758015.575497:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758015.575501:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758015.575505:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655040512:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758015.575512:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655040640:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758015.575534:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758015.575546:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758015/real 1787758015] req@ffff8c05b2d9aa00 x1874588655040640/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758015.575560:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655040640/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758015.575579:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758015.575583:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758015.575587:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655040640:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758015.702045:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758016.150258:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758016.726011:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758017.174114:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758017.750013:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758018.198137:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758018.773974:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758019.222034:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758019.797959:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758020.246097:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758020.693991:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758020.694006:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758020.694012:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758020.694025:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11077 00000100:00080000:1.0:1787758020.694039:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758020.694050:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758020.694105:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655040768/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758020.694146:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758020.694150:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758020.694152:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758020.694157:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11077 00000100:00080000:1.0:1787758020.694162:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758020.694168:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758020.694180:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758020.694193:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655040768:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758020.694201:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588655040896/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758020.694214:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758020.694219:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758020.694221:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758020.694226:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11077 00000100:00080000:1.0:1787758020.694230:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758020.694236:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758020.694241:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758020.694256:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655041024/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758020.694268:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787758020.694272:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758020.694276:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758020.694280:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758020.694286:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11077 00000100:00080000:1.0:1787758020.694291:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758020.694297:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758020.694304:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758020.694330:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655041152/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758020.694342:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787758020.694342:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758020/real 1787758020] req@ffff8c05b2d98380 x1874588655040768/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758020.694351:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758020.694356:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758020.694364:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11077 00000100:00080000:1.0:1787758020.694369:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787758020.694373:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655040768/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758020.694377:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758020.694385:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787758020.694393:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787758020.694400:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655041280/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758020.694404:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758020.694413:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655040768:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758020.694418:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758020.694424:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758020.694426:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758020.694432:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11077 00000100:00080000:1.0:1787758020.694436:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758020.694441:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758020.694446:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655040896:192.168.201.107@tcp:38:kworker.0 00000008:00100000:1.0:1787758020.694447:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758020.694473:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655041408/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787758020.694479:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758020.694491:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758020/real 1787758020] req@ffff8c05b2d9a300 x1874588655040896/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758020.694517:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758020.694523:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787758020.694524:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588655040896/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758020.694528:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758020.694538:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11077 00000100:00080000:3.0:1787758020.694544:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758020.694545:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758020.694554:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787758020.694555:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758020.694564:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758020.694565:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655040896:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758020.694578:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655041024:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758020.694581:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655041536/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758020.694592:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758020.694595:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758020.694597:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758020.694600:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11077 00000100:00080000:1.0:1787758020.694604:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787758020.694607:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758020.694611:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758020.694617:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758020.694622:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758020/real 1787758020] req@ffff8c05b2d9aa00 x1874588655041024/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758020.694632:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655041664/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787758020.694635:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655041024/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758020.694659:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758020.694665:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758020.694670:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655041024:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758020.694675:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655041152:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758020.694689:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758020.694696:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758020/real 1787758020] req@ffff8c05b2d9bb80 x1874588655041152/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758020.694703:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655041152/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758020.694714:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758020.694716:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758020.694718:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655041152:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758020.694722:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655041280:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758020.694732:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758020.694738:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758020/real 1787758020] req@ffff8c05b2d98700 x1874588655041280/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758020.694745:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655041280/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758020.694751:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758020.694753:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758020.694755:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655041280:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758020.694769:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655041408:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758020.694780:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758020.694787:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758020/real 1787758020] req@ffff8c05b2d9b100 x1874588655041408/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758020.694794:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655041408/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758020.694799:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758020.694801:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758020.694803:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655041408:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758020.694807:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655041536:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758020.694817:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758020.694822:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758020/real 1787758020] req@ffff8c05b2d99880 x1874588655041536/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758020.694830:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655041536/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758020.694835:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758020.694837:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758020.694839:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655041536:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758020.694842:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655041664:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758020.694851:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758020.694857:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758020/real 1787758020] req@ffff8c05b2d99f80 x1874588655041664/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758020.694865:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655041664/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758020.694870:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758020.694871:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758020.694873:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655041664:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758020.821984:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758021.270036:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758021.845950:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758022.294117:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758022.869998:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758023.318054:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758023.893959:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758024.342078:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758024.917940:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758025.366012:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758025.814014:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758025.814021:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758025.814025:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758025.814033:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11082 00000100:00080000:0.0:1787758025.814043:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758025.814048:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758025.814081:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655041792/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758025.814103:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758025.814105:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758025.814107:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758025.814111:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11082 00000100:00080000:0.0:1787758025.814113:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758025.814117:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758025.814125:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758025.814136:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655041792:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787758025.814138:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655041920/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758025.814153:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758025.814156:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758025.814157:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758025.814160:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11082 00000100:00080000:0.0:1787758025.814163:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758025.814165:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758025.814170:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758025.814180:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588655042048/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758025.814188:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758025.814191:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758025.814192:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758025.814195:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11082 00000100:00080000:0.0:1787758025.814198:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758025.814201:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758025.814205:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787758025.814209:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758025.814214:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588655042176/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758025.814236:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758025.814239:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758025.814241:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758025.814244:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11082 00000100:00080000:0.0:1787758025.814247:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758025.814251:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758025.814256:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758025.814259:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758025/real 1787758025] req@ffff8c05a9f76680 x1874588655041792/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758025.814267:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588655042304/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758025.814276:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758025.814278:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655041792/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758025.814279:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758025.814283:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758025.814290:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11082 00000100:00080000:0.0:1787758025.814294:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758025.814296:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758025.814298:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758025.814303:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787758025.814303:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758025.814310:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655041792:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787758025.814315:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655042432/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758025.814326:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758025.814329:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758025.814331:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758025.814334:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655041920:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758025.814337:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11082 00000100:00080000:0.0:1787758025.814342:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758025.814344:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758025.814347:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758025.814357:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655042560/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787758025.814364:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758025.814369:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758025.814373:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758025.814374:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758025.814379:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11082 00000100:00100000:1.0:1787758025.814381:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758025/real 1787758025] req@ffff8c05a9f77800 x1874588655041920/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758025.814383:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758025.814391:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758025.814398:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758025.814403:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655041920/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758025.814409:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655042688/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758025.814415:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758025.814423:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758025.814427:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655041920:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758025.814433:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655042048:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758025.814452:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758025.814464:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758025/real 1787758025] req@ffff8c05a9f75c00 x1874588655042048/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758025.814484:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588655042048/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758025.814494:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758025.814497:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758025.814500:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655042048:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758025.814504:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655042176:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758025.814522:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758025.814530:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758025/real 1787758025] req@ffff8c05a9f76a00 x1874588655042176/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758025.814539:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588655042176/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758025.814546:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758025.814548:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758025.814550:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655042176:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758025.814554:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655042304:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758025.814565:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758025.814572:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758025/real 1787758025] req@ffff8c05a9f77b80 x1874588655042304/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758025.814581:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588655042304/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758025.814591:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758025.814594:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758025.814597:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655042304:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758025.814603:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655042432:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758025.814616:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758025.814627:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758025/real 1787758025] req@ffff8c05a9f76300 x1874588655042432/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758025.814637:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655042432/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758025.814645:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758025.814647:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758025.814649:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655042432:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758025.814668:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655042560:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758025.814690:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758025.814697:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758025/real 1787758025] req@ffff8c05a9f74380 x1874588655042560/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758025.814704:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655042560/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758025.814709:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758025.814711:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758025.814713:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655042560:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758025.814717:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655042688:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758025.814728:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758025.814733:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758025/real 1787758025] req@ffff8c05a9f74a80 x1874588655042688/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758025.814740:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655042688/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758025.814745:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758025.814747:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758025.814749:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655042688:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758025.941942:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758026.390113:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758026.965951:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758027.414063:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758027.989953:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758028.438164:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758029.013947:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758029.462034:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758030.037949:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758030.486151:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758030.934039:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758030.934051:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758030.934057:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758030.934087:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11087 00000100:00080000:3.0:1787758030.934098:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758030.934106:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758030.934177:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655042816/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758030.934214:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758030.934217:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758030.934220:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758030.934225:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11087 00000100:00080000:3.0:1787758030.934229:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758030.934253:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758030.934265:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758030.934269:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655042816:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787758030.934285:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655042944/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758030.934297:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758030.934300:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758030.934301:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758030.934306:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11087 00000100:00080000:3.0:1787758030.934311:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758030.934315:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758030.934319:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758030.934330:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655043072/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758030.934341:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758030.934344:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758030.934345:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787758030.934351:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758030.934352:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11087 00000100:00080000:3.0:1787758030.934377:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758030.934382:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758030.934387:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758030.934397:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655043200/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758030.934408:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758030.934411:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758030.934415:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758030.934416:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758030/real 1787758030] req@ffff8c05a9f7bb80 x1874588655042816/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758030.934422:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11087 00000100:00080000:3.0:1787758030.934429:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758030.934434:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758030.934440:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758030.934444:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655042816/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758030.934456:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655043328/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758030.934457:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758030.934467:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758030.934473:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655042816:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758030.934474:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758030.934481:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758030.934484:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758030.934488:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11087 00000100:00080000:3.0:1787758030.934492:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758030.934496:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758030.934502:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758030.934504:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655042944:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758030.934518:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655043456/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758030.934527:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758030.934530:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1787758030.934534:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758030.934534:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758030.934543:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11087 00000100:00080000:3.0:1787758030.934549:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758030.934554:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758030.934558:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758030/real 1787758030] req@ffff8c05a9f79f80 x1874588655042944/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787758030.934573:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758030.934590:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655043584/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758030.934592:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655042944/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758030.934618:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758030.934625:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787758030.934626:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758030.934630:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655042944:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758030.934631:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758030.934635:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758030.934640:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655043072:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758030.934644:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11087 00000100:00080000:3.0:1787758030.934660:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787758030.934663:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758030.934669:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758030.934675:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758030.934676:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758030/real 1787758030] req@ffff8c05a9f79180 x1874588655043072/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758030.934688:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655043072/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758030.934693:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655043712/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758030.934699:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758030.934708:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758030.934711:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655043072:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758030.934716:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655043200:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758030.934729:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758030.934737:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758030/real 1787758030] req@ffff8c05a9f7b100 x1874588655043200/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758030.934750:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655043200/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758030.934756:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758030.934758:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758030.934760:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655043200:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758030.934763:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655043328:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758030.934775:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758030.934781:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758030/real 1787758030] req@ffff8c05a9f79500 x1874588655043328/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758030.934789:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655043328/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758030.934794:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758030.934795:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758030.934797:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655043328:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758030.934809:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655043456:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758030.934819:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758030.934825:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758030/real 1787758030] req@ffff8c05a9f78e00 x1874588655043456/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758030.934832:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655043456/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758030.934842:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758030.934844:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758030.934846:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655043456:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758030.934849:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655043584:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758030.934858:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758030.934863:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758030/real 1787758030] req@ffff8c05a9f7b800 x1874588655043584/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758030.934870:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655043584/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758030.934914:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758030.934917:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758030.934919:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655043584:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758030.934924:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655043712:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758030.934936:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758030.934944:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758030/real 1787758030] req@ffff8c05a9f7ad80 x1874588655043712/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758030.934952:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655043712/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758030.934957:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758030.934959:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758030.934962:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655043712:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758031.061987:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758031.510044:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758032.085967:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758032.534127:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758033.109953:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758033.558079:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758034.133992:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758034.582122:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758035.157990:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758035.606009:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758036.053930:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758036.053939:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758036.053943:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758036.053950:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11092 00000100:00080000:3.0:1787758036.053958:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758036.053963:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758036.053993:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655043840/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758036.054013:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758036.054017:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758036.054019:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758036.054023:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11092 00000100:00080000:3.0:1787758036.054027:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758036.054032:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758036.054056:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758036.054061:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655043840:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787758036.054074:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655043968/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758036.054089:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758036.054093:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758036.054095:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758036.054100:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11092 00000100:00080000:3.0:1787758036.054105:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758036.054110:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758036.054116:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787758036.054119:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758036.054136:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655044096/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758036.054148:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758036.054153:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787758036.054157:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758036/real 1787758036] req@ffff8c05a9f7b480 x1874588655043840/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758036.054157:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758036.054169:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11092 00000100:00080000:3.0:1787758036.054176:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758036.054179:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655043840/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758036.054184:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758036.054193:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787758036.054196:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758036.054201:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758036.054205:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655043840:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787758036.054208:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655044224/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758036.054218:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758036.054220:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758036.054222:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758036.054225:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11092 00000100:00100000:1.0:1787758036.054229:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655043968:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758036.054229:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758036.054235:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758036.054240:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758036.054250:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655044352/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787758036.054254:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758036.054264:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758036.054269:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758036/real 1787758036] req@ffff8c05a9f78380 x1874588655043968/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758036.054269:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758036.054273:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758036.054281:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11092 00000100:00080000:3.0:1787758036.054288:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758036.054291:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758036.054295:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655043968/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787758036.054297:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787758036.054307:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758036.054310:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758036.054312:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655043968:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758036.054317:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655044096:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758036.054322:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655044480/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758036.054330:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758036.054334:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1787758036.054335:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758036.054336:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758036.054342:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11092 00000100:00080000:3.0:1787758036.054345:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758036.054347:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758036/real 1787758036] req@ffff8c05a9f78a80 x1874588655044096/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758036.054351:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758036.054360:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758036.054365:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655044096/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758036.054371:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655044608/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758036.054375:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758036.054384:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787758036.054390:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758036.054391:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655044096:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758036.054397:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758036.054400:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758036.054401:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655044224:192.168.201.107@tcp:8:kworker.0 00000100:00080000:3.0:1787758036.054407:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11092 00000100:00080000:3.0:1787758036.054411:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758036.054413:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758036.054417:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787758036.054422:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758036.054428:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655044736/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758036.054433:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758036/real 1787758036] req@ffff8c05a9f7a680 x1874588655044224/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758036.054452:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655044224/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758036.054459:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758036.054462:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758036.054464:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655044224:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758036.054474:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655044352:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758036.054485:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758036.054492:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758036/real 1787758036] req@ffff8c05a9f7aa00 x1874588655044352/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758036.054503:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655044352/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758036.054509:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758036.054511:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758036.054513:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655044352:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758036.054516:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655044480:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758036.054532:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758036.054543:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758036/real 1787758036] req@ffff8c05a9f7a300 x1874588655044480/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758036.054556:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655044480/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758036.054565:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758036.054569:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758036.054572:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655044480:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758036.054579:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655044608:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758036.054594:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758036.054602:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758036/real 1787758036] req@ffff8c05a9f79880 x1874588655044608/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758036.054609:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655044608/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758036.054619:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758036.054621:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758036.054623:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655044608:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758036.054627:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655044736:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758036.054637:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758036.054643:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758036/real 1787758036] req@ffff8c05a9f78700 x1874588655044736/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758036.054652:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655044736/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758036.054660:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758036.054663:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758036.054666:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655044736:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758036.181944:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758036.630072:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758037.205945:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758037.654001:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758038.229978:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758038.678129:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758039.254018:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758039.702132:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758040.278082:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758040.726206:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758041.174067:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758041.174075:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758041.174079:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758041.174088:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11097 00000100:00080000:0.0:1787758041.174098:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758041.174104:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758041.174138:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588655044864/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758041.174158:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758041.174160:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758041.174162:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758041.174165:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11097 00000100:00080000:0.0:1787758041.174167:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758041.174170:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758041.174176:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758041.174186:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588655044992/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758041.174195:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758041.174198:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758041.174199:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758041.174205:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11097 00000100:00080000:0.0:1787758041.174211:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758041.174229:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655044864:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758041.174236:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758041.174249:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758041.174260:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588655045120/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758041.174274:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758041.174277:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758041.174280:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758041.174283:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11097 00000100:00080000:0.0:1787758041.174286:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758041.174288:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758041.174292:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758041.174306:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588655045248/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758041.174312:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758041.174315:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758041.174316:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758041.174320:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11097 00000100:00080000:0.0:1787758041.174342:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758041.174345:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758041.174350:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787758041.174363:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758041.174364:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588655045376/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758041.174382:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758041.174386:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758041.174389:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758041.174395:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11097 00000100:00080000:0.0:1787758041.174418:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758041.174421:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758041.174425:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758041.174434:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655045504/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758041.174442:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758041.174445:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758041.174446:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758041.174450:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11097 00000100:00080000:0.0:1787758041.174456:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758041.174458:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758041/real 1787758041] req@ffff8c05a9f74e00 x1874588655044864/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758041.174464:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758041.174469:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758041.174492:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588655045632/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758041.174529:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758041.174534:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758041.174536:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758041.174540:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11097 00000100:00100000:1.0:1787758041.174547:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588655044864/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758041.174549:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758041.174564:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758041.174579:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758041.174596:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655045760/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758041.174649:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758041.174656:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758041.174661:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655044864:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758041.174689:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655044992:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758041.174713:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758041.174725:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758041/real 1787758041] req@ffff8c05a9f76d80 x1874588655044992/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758041.174734:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588655044992/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758041.174741:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758041.174744:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758041.174746:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655044992:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758041.174751:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655045120:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758041.174764:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758041.174771:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758041/real 1787758041] req@ffff8c05a9f77100 x1874588655045120/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758041.174779:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588655045120/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758041.174785:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758041.174788:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758041.174790:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655045120:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758041.174795:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655045248:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758041.174808:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758041.174814:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758041/real 1787758041] req@ffff8c05a9f75180 x1874588655045248/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758041.174823:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588655045248/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758041.174829:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758041.174831:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758041.174833:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655045248:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758041.174837:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655045376:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758041.174849:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758041.174866:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758041/real 1787758041] req@ffff8c05a9f74000 x1874588655045376/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758041.174947:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588655045376/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758041.174997:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758041.175000:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758041.175002:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655045376:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758041.175007:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655045504:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758041.175025:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758041.175034:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758041/real 1787758041] req@ffff8c05a9f75880 x1874588655045504/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758041.175044:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655045504/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758041.175050:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758041.175052:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758041.175054:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655045504:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758041.175058:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655045632:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758041.175070:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758041.175076:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758041/real 1787758041] req@ffff8c05a9f75500 x1874588655045632/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758041.175084:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588655045632/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758041.175090:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758041.175092:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758041.175094:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655045632:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758041.175097:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655045760:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758041.175108:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758041.175115:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758041/real 1787758041] req@ffff8c05a9f74a80 x1874588655045760/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758041.175122:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655045760/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758041.175133:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758041.175136:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758041.175138:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655045760:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758041.302026:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758041.750083:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758042.326087:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758042.774283:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758043.350072:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758043.798118:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758044.374042:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758044.822151:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758045.398051:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758045.846217:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758046.294061:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758046.294078:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758046.294087:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758046.294108:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11102 00000100:00080000:1.0:1787758046.294130:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758046.294146:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758046.294231:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655045888/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758046.294282:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758046.294289:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758046.294293:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758046.294304:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11102 00000100:00080000:1.0:1787758046.294329:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758046.294341:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758046.294351:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655045888:192.168.201.107@tcp:250:kworker.0 00000008:00100000:1.0:1787758046.294365:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758046.294401:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655046016/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758046.294420:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758046.294427:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758046.294430:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758046.294436:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11102 00000100:00080000:1.0:1787758046.294444:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758046.294450:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758046.294458:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787758046.294473:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758046.294482:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655046144/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758046.294500:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758046.294505:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758046.294507:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758046.294523:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11102 00000100:00080000:1.0:1787758046.294528:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758046.294532:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758046.294539:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758046.294557:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655046272/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787758046.294567:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758046/real 1787758046] req@ffff8c05b2d98000 x1874588655045888/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758046.294576:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758046.294591:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758046.294598:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758046.294608:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11102 00000100:00080000:1.0:1787758046.294614:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787758046.294623:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655045888/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758046.294624:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758046.294652:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787758046.294661:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758046.294672:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758046.294678:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655046400/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787758046.294684:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655045888:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758046.294716:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758046.294723:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758046.294729:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787758046.294735:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655046016:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758046.294747:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11102 00000100:00080000:1.0:1787758046.294757:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758046.294767:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758046.294779:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787758046.294779:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758046.294805:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758046/real 1787758046] req@ffff8c05b2d9b480 x1874588655046016/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758046.294812:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655046528/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787758046.294848:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655046016/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758046.294849:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758046.294939:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758046.294945:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758046.294956:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655046016:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758046.294957:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758046.294966:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787758046.294977:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655046144:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758046.294978:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11102 00000100:00080000:1.0:1787758046.294993:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758046.295003:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758046.295014:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787758046.295026:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758046.295042:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655046656/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787758046.295052:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758046/real 1787758046] req@ffff8c05b2d99f80 x1874588655046144/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758046.295071:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758046.295089:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787758046.295093:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655046144/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758046.295097:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758046.295119:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11102 00000100:00080000:3.0:1787758046.295122:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758046.295135:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758046.295135:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758046.295147:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655046144:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758046.295157:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758046.295167:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655046272:192.168.201.107@tcp:8:kworker.0 00000008:00100000:1.0:1787758046.295177:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787758046.295207:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758046.295215:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655046784/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787758046.295231:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758046/real 1787758046] req@ffff8c05b2d99880 x1874588655046272/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758046.295260:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655046272/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758046.295272:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758046.295277:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758046.295281:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655046272:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758046.295289:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655046400:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758046.295309:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758046.295321:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758046/real 1787758046] req@ffff8c05b2d9b100 x1874588655046400/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758046.295335:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655046400/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758046.295345:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758046.295348:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758046.295352:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655046400:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758046.295374:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655046528:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758046.295395:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758046.295408:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758046/real 1787758046] req@ffff8c05b2d98700 x1874588655046528/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758046.295423:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655046528/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758046.295433:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758046.295437:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758046.295440:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655046528:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758046.295447:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655046656:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758046.295465:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758046.295476:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758046/real 1787758046] req@ffff8c05b2d9bb80 x1874588655046656/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758046.295491:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655046656/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758046.295505:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758046.295511:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758046.295525:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655046656:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758046.295536:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655046784:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758046.295572:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758046.295595:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758046/real 1787758046] req@ffff8c05b2d9aa00 x1874588655046784/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758046.295620:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655046784/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758046.295650:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758046.295655:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758046.295660:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655046784:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758046.422003:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758046.870243:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758047.446027:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758047.894109:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758048.470034:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758048.918227:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758049.494069:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758049.942103:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758050.518099:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758050.966371:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758051.414067:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758051.414084:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758051.414092:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758051.414109:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11107 00000100:00080000:3.0:1787758051.414127:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758051.414139:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758051.414226:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655046912/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758051.414271:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758051.414276:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758051.414279:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758051.414285:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11107 00000100:00080000:3.0:1787758051.414290:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758051.414296:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758051.414310:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758051.414328:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655047040/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758051.414345:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655046912:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758051.414348:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758051.414362:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758051.414365:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758051.414371:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11107 00000100:00080000:3.0:1787758051.414376:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758051.414380:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758051.414387:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758051.414402:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655047168/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758051.414416:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758051.414421:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758051.414424:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758051.414429:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11107 00000100:00080000:3.0:1787758051.414434:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758051.414438:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758051.414444:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758051.414457:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655047296/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758051.414470:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758051.414475:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758051.414477:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758051.414481:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11107 00000100:00080000:3.0:1787758051.414487:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787758051.414494:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758051.414502:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758051.414511:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758051.414529:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655047424/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758051.414541:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758051.414545:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758051.414547:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758051.414552:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11107 00000100:00080000:3.0:1787758051.414557:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758051.414562:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758051.414568:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758051.414578:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758051/real 1787758051] req@ffff8c05a9f78700 x1874588655046912/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758051.414599:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655047552/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758051.414620:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758051.414627:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655046912/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758051.414627:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758051.414634:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758051.414647:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11107 00000100:00080000:3.0:1787758051.414657:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758051.414668:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758051.414677:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787758051.414679:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787758051.414696:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655047680/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758051.414698:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758051.414720:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655046912:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758051.414722:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758051.414736:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758051.414742:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758051.414757:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11107 00000100:00080000:3.0:1787758051.414762:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758051.414766:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758051.414797:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758051.414813:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655047040:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758051.414819:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655047808/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787758051.414919:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758051.414949:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758051/real 1787758051] req@ffff8c05a9f79880 x1874588655047040/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758051.414980:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655047040/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758051.414995:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758051.415002:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758051.415009:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655047040:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758051.415022:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655047168:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758051.415066:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758051.415090:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758051/real 1787758051] req@ffff8c05a9f7a300 x1874588655047168/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758051.415117:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655047168/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758051.415129:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758051.415133:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758051.415137:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655047168:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758051.415146:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655047296:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758051.415177:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758051.415191:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758051/real 1787758051] req@ffff8c05a9f7aa00 x1874588655047296/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758051.415206:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655047296/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758051.415226:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758051.415230:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758051.415234:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655047296:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758051.415240:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655047424:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758051.415260:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758051.415271:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758051/real 1787758051] req@ffff8c05a9f7a680 x1874588655047424/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758051.415285:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655047424/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758051.415296:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758051.415299:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758051.415303:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655047424:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758051.415309:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655047552:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758051.415328:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758051.415339:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758051/real 1787758051] req@ffff8c05a9f78a80 x1874588655047552/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758051.415353:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655047552/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758051.415362:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758051.415365:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758051.415370:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655047552:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758051.415377:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655047680:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758051.415397:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758051.415408:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758051/real 1787758051] req@ffff8c05a9f78380 x1874588655047680/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758051.415421:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655047680/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758051.415432:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758051.415435:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758051.415439:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655047680:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758051.415465:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655047808:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758051.415485:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758051.415496:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758051/real 1787758051] req@ffff8c05a9f7b480 x1874588655047808/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758051.415510:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655047808/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758051.415521:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758051.415524:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758051.415528:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655047808:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758051.542023:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758051.990182:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758052.566005:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758053.014142:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758053.590142:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758054.038173:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758054.614002:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758055.062186:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758055.638009:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758056.086136:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758056.533962:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758056.533974:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758056.533979:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758056.533990:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11112 00000100:00080000:0.0:1787758056.534001:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758056.534007:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758056.534041:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655047936/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758056.534065:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758056.534068:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758056.534069:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758056.534072:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11112 00000100:00080000:0.0:1787758056.534075:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758056.534077:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758056.534084:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758056.534093:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655048064/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758056.534101:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758056.534104:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758056.534105:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758056.534110:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11112 00000100:00100000:1.0:1787758056.534111:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655047936:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758056.534113:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758056.534120:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758056.534124:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758056.534137:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588655048192/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758056.534148:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758056.534152:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758056.534154:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758056.534159:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11112 00000100:00080000:0.0:1787758056.534166:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758056.534170:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:1.0:1787758056.534175:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:0.0:1787758056.534177:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758056.534189:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588655048320/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758056.534197:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758056.534199:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758056.534201:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758056.534204:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11112 00000100:00080000:0.0:1787758056.534207:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758056.534211:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758056/real 1787758056] req@ffff8c05a9f74380 x1874588655047936/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758056.534212:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758056.534219:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758056.534233:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588655048448/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758056.534236:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655047936/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758056.534264:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758056.534266:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758056.534269:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787758056.534271:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758056.534274:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758056.534276:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655047936:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758056.534280:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11112 00000100:00080000:0.0:1787758056.534286:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758056.534288:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758056.534291:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758056.534298:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655048576/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758056.534305:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758056.534306:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655048064:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758056.534309:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758056.534313:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758056.534319:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11112 00000100:00080000:0.0:1787758056.534323:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758056.534327:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758056.534333:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787758056.534344:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758056.534349:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655048704/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758056.534361:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758056.534364:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758056/real 1787758056] req@ffff8c05a9f76300 x1874588655048064/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758056.534367:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758056.534371:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758056.534380:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11112 00000100:00080000:0.0:1787758056.534387:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758056.534390:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758056.534394:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758056.534404:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655048832/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758056.534408:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655048064/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758056.534427:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758056.534432:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758056.534436:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655048064:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758056.534444:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655048192:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758056.534462:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758056.534471:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758056/real 1787758056] req@ffff8c05a9f77b80 x1874588655048192/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758056.534484:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588655048192/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758056.534489:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758056.534491:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758056.534493:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655048192:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758056.534497:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655048320:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758056.534507:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758056.534513:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758056/real 1787758056] req@ffff8c05a9f76a00 x1874588655048320/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758056.534520:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588655048320/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758056.534526:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758056.534528:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758056.534530:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655048320:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758056.534533:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655048448:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758056.534542:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758056.534548:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758056/real 1787758056] req@ffff8c05a9f75c00 x1874588655048448/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758056.534555:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588655048448/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758056.534565:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758056.534567:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758056.534569:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655048448:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758056.534581:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655048576:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758056.534592:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758056.534597:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758056/real 1787758056] req@ffff8c05a9f77800 x1874588655048576/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758056.534604:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655048576/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758056.534610:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758056.534612:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758056.534613:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655048576:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758056.534617:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655048704:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758056.534627:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758056.534632:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758056/real 1787758056] req@ffff8c05a9f76680 x1874588655048704/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758056.534639:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655048704/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758056.534644:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758056.534646:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758056.534648:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655048704:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758056.534654:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655048832:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758056.534672:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758056.534699:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758056/real 1787758056] req@ffff8c05a9f74380 x1874588655048832/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758056.534713:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655048832/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758056.534723:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758056.534726:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758056.534728:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655048832:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758056.661971:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758057.110158:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758057.686009:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758058.134125:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758058.709963:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758059.158215:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758059.734016:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758060.182116:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758060.758032:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758061.206208:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758061.654038:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758061.654049:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758061.654054:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758061.654063:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11118 00000100:00080000:0.0:1787758061.654075:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758061.654081:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758061.654125:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655048960/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758061.654156:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758061.654160:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758061.654162:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758061.654168:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11118 00000100:00080000:0.0:1787758061.654172:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758061.654175:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758061.654184:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758061.654198:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588655049088/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758061.654208:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758061.654212:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758061.654214:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758061.654218:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11118 00000100:00080000:0.0:1787758061.654223:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758061.654230:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655048960:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758061.654231:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758061.654241:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758061.654255:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655049216/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758061.654265:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758061.654268:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758061.654270:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758061.654274:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11118 00000100:00080000:0.0:1787758061.654277:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758061.654280:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758061.654286:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758061.654300:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588655049344/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758061.654313:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787758061.654320:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758061.654322:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758061.654327:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758061.654338:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11118 00000100:00080000:0.0:1787758061.654344:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758061.654348:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758061.654354:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758061.654378:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588655049472/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758061.654388:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758061/real 1787758061] req@ffff8c05a9f74a80 x1874588655048960/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758061.654399:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758061.654408:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758061.654411:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758061.654414:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655048960/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758061.654424:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11118 00000100:00080000:1.0:1787758061.654432:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758061.654435:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758061.654440:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787758061.654451:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758061.654451:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655048960:192.168.201.107@tcp:250:kworker.0 00000008:00100000:0.0:1787758061.654465:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758061.654467:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655049088:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758061.654491:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588655049600/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787758061.654502:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758061.654504:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758061.654509:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758061.654512:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758061.654519:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11118 00000100:00100000:1.0:1787758061.654521:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758061/real 1787758061] req@ffff8c05a9f75500 x1874588655049088/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758061.654527:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758061.654538:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758061.654544:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588655049088/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787758061.654546:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787758061.654563:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787758061.654565:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588655049728/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758061.654571:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758061.654582:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655049088:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758061.654587:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758061.654614:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758061.654617:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758061.654642:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11118 00000100:00080000:0.0:1787758061.654651:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758061.654658:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758061.654668:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758061.654669:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655049216:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758061.654700:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588655049856/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787758061.654722:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758061.654750:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758061/real 1787758061] req@ffff8c05a9f75880 x1874588655049216/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758061.654770:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655049216/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758061.654781:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758061.654787:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758061.654794:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655049216:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758061.654804:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655049344:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758061.654827:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758061.654840:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758061/real 1787758061] req@ffff8c05a9f74000 x1874588655049344/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758061.654863:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588655049344/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758061.654874:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758061.654878:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758061.654884:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655049344:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758061.654963:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655049472:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758061.654991:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758061.655009:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758061/real 1787758061] req@ffff8c05a9f75180 x1874588655049472/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758061.655040:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588655049472/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758061.655049:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758061.655054:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758061.655058:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655049472:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758061.655074:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655049600:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758061.655096:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758061.655125:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758061/real 1787758061] req@ffff8c05a9f77100 x1874588655049600/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758061.655149:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588655049600/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758061.655172:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758061.655176:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758061.655179:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655049600:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758061.655186:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655049728:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758061.655203:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758061.655215:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758061/real 1787758061] req@ffff8c05a9f76d80 x1874588655049728/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758061.655230:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588655049728/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758061.655237:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758061.655242:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758061.655246:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655049728:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758061.655305:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655049856:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758061.655343:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758061.655379:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758061/real 1787758061] req@ffff8c05a9f74e00 x1874588655049856/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758061.655392:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588655049856/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758061.655400:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758061.655403:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758061.655407:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655049856:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758061.782008:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758062.230109:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758062.806007:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758063.254251:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758063.829954:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758064.278020:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758064.853964:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758065.302233:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758065.878027:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758066.326058:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758066.773982:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758066.773994:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758066.773998:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758066.774010:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11123 00000100:00080000:0.0:1787758066.774021:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758066.774029:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758066.774074:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588655049984/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758066.774099:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758066.774102:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758066.774103:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758066.774107:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11123 00000100:00080000:0.0:1787758066.774110:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758066.774112:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758066.774120:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758066.774130:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588655050112/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758066.774143:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758066.774146:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758066.774147:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758066.774152:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11123 00000100:00080000:0.0:1787758066.774154:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758066.774157:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758066.774160:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758066.774167:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588655050240/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758066.774176:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758066.774179:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758066.774181:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758066.774184:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11123 00000100:00080000:0.0:1787758066.774186:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758066.774194:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758066.774197:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655049984:192.168.201.107@tcp:250:kworker.0 00000008:00100000:0.0:1787758066.774199:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758066.774214:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588655050368/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758066.774224:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758066.774227:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758066.774229:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758066.774232:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11123 00000100:00080000:0.0:1787758066.774237:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758066.774242:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758066.774246:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758066.774254:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588655050496/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758066.774261:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758066.774264:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758066.774265:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758066.774269:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11123 00000100:00080000:0.0:1787758066.774272:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758066.774273:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758066.774276:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758066.774285:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655050624/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758066.774291:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758066.774294:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758066.774296:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758066.774301:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11123 00000100:00100000:1.0:1787758066.774304:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655050112:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758066.774308:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758066.774317:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758066.774320:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758066.774335:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588655050752/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758066.774340:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655050240:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758066.774366:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758066.774369:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758066.774373:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758066.774378:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11123 00000100:00080000:0.0:1787758066.774380:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758066.774384:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758066.774388:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758066.774391:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655050368:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758066.774403:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655050880/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758066.774421:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655050496:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758066.774452:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655050624:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758066.774477:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655050752:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758066.774514:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655050880:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758066.901966:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758067.350055:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758067.925958:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758068.374177:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758068.950028:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758069.398220:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787758069.846300:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.107:988 00000400:00000100:2.0:1787758069.846375:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.107@tcp at host 192.168.201.107 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787758069.846408:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787758069.846503:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787758069.846524:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787758069.846546:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787758069.846571:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787758069.846595:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:1.0:1787758069.846601:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758066/real 1787758069] req@ffff8c05a9f75180 x1874588655050368/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787758069.846620:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787758069.846649:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:1.0:1787758069.846679:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588655050368/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758069.846718:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758069.846730:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758069.846740:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655050368:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758069.846758:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758066/real 1787758069] req@ffff8c05a9f74000 x1874588655050496/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758069.846788:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588655050496/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758069.846806:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758069.846813:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758069.846821:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655050496:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758069.846836:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758066/real 1787758069] req@ffff8c05a9f75880 x1874588655050624/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758069.846862:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655050624/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758069.846933:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758069.846941:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758069.846949:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655050624:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758069.846964:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758066/real 1787758069] req@ffff8c05a9f75500 x1874588655050752/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758069.846991:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588655050752/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758069.847007:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758069.847014:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758069.847021:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655050752:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758069.847036:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758066/real 1787758069] req@ffff8c05a9f74e00 x1874588655049984/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758069.847069:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588655049984/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758069.847088:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758069.847095:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758069.847103:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655049984:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758069.847117:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758066/real 1787758069] req@ffff8c05a9f76d80 x1874588655050112/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758069.847143:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588655050112/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758069.847158:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758069.847165:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758069.847171:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655050112:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758069.847186:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758066/real 1787758069] req@ffff8c05a9f77100 x1874588655050240/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758069.847209:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588655050240/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758069.847224:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758069.847230:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758069.847236:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655050240:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758069.847301:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758066/real 1787758069] req@ffff8c05a9f74a80 x1874588655050880/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758069.847328:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655050880/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758069.847344:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758069.847352:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758069.847358:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655050880:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758069.974011:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758070.422150:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758070.998041:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758071.446176:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758071.894021:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758071.894032:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758071.894036:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758071.894047:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11128 00000100:00080000:0.0:1787758071.894058:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758071.894066:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758071.894118:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655051008/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758071.894150:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758071.894154:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758071.894156:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758071.894161:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11128 00000100:00080000:0.0:1787758071.894175:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758071.894180:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758071.894189:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758071.894204:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655051136/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758071.894215:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758071.894221:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758071.894225:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758071.894225:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655051008:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758071.894233:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11128 00000100:00080000:0.0:1787758071.894242:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758071.894246:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758071.894250:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758071.894261:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655051264/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758071.894271:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758071.894274:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758071.894276:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758071.894280:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11128 00000100:00080000:0.0:1787758071.894283:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758071.894286:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758071.894290:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758071.894300:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588655051392/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758071.894308:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758071.894313:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758071.894315:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758071.894319:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11128 00000100:00080000:0.0:1787758071.894323:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758071.894326:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758071.894330:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787758071.894336:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758071.894345:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588655051520/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758071.894355:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758071.894359:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758071.894361:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758071.894366:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11128 00000100:00080000:0.0:1787758071.894369:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758071.894372:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758071.894376:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758071.894389:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588655051648/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758071.894402:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758071/real 1787758071] req@ffff8c05a9f74380 x1874588655051008/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758071.894459:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655051008/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758071.894464:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758071.894478:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758071.894485:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758071.894499:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11128 00000100:00080000:1.0:1787758071.894500:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758071.894510:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758071.894514:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787758071.894522:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758071.894528:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655051008:192.168.201.107@tcp:250:kworker.0 00000008:00100000:0.0:1787758071.894534:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758071.894571:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655051776/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758071.894575:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655051136:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758071.894604:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758071.894609:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758071.894614:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758071.894621:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11128 00000100:00080000:0.0:1787758071.894630:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787758071.894633:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758071.894642:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758071.894653:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758071.894663:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758071/real 1787758071] req@ffff8c05a9f76680 x1874588655051136/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758071.894676:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655051904/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758071.894735:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655051136/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758071.894751:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758071.894758:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758071.894764:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655051136:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758071.894774:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655051264:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758071.894806:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758071.894824:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758071/real 1787758071] req@ffff8c05a9f77800 x1874588655051264/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758071.894844:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655051264/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758071.894857:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758071.894862:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758071.894867:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655051264:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758071.894928:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655051392:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758071.894961:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758071.894978:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758071/real 1787758071] req@ffff8c05a9f75c00 x1874588655051392/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758071.894994:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588655051392/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758071.895013:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758071.895018:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758071.895022:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655051392:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758071.895031:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655051520:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758071.895058:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758071.895076:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758071/real 1787758071] req@ffff8c05a9f76a00 x1874588655051520/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758071.895094:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588655051520/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758071.895107:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758071.895111:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758071.895116:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655051520:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758071.895125:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655051648:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758071.895157:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758071.895174:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758071/real 1787758071] req@ffff8c05a9f77b80 x1874588655051648/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758071.895192:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588655051648/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758071.895204:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758071.895208:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758071.895213:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655051648:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758071.895254:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655051776:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758071.895285:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758071.895304:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758071/real 1787758071] req@ffff8c05a9f76300 x1874588655051776/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758071.895322:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655051776/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758071.895334:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758071.895338:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758071.895343:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655051776:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758071.895372:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655051904:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758071.895405:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758071.895442:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758071/real 1787758071] req@ffff8c05a9f74380 x1874588655051904/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758071.895468:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655051904/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758071.895480:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758071.895485:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758071.895489:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655051904:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758072.022074:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758072.470139:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758073.045974:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758073.494100:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758074.069939:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758074.518209:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758075.093979:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758075.542286:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758076.118067:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758076.566207:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758077.013977:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758077.013990:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758077.013995:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758077.014007:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11133 00000100:00080000:0.0:1787758077.014020:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758077.014028:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758077.014075:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655052032/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758077.014111:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758077.014116:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758077.014119:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758077.014124:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11133 00000100:00080000:0.0:1787758077.014129:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758077.014134:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758077.014144:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758077.014159:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588655052160/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758077.014172:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758077.014176:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758077.014178:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758077.014182:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11133 00000100:00080000:0.0:1787758077.014186:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758077.014190:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758077.014196:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758077.014207:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588655052288/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758077.014221:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758077.014225:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758077.014228:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758077.014238:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11133 00000100:00100000:1.0:1787758077.014242:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655052032:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758077.014243:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758077.014256:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758077.014266:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758077.014280:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588655052416/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758077.014294:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758077.014299:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758077.014302:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758077.014306:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11133 00000100:00080000:0.0:1787758077.014353:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758077.014358:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758077.014365:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787758077.014366:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758077.014390:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588655052544/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758077.014404:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758077.014409:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758077.014411:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758077.014418:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11133 00000100:00080000:0.0:1787758077.014424:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758077.014431:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758077/real 1787758077] req@ffff8c05a9f74a80 x1874588655052032/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758077.014435:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758077.014448:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758077.014473:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655052672/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758077.014480:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655052032/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758077.014516:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758077.014523:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758077.014524:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758077.014533:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758077.014539:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787758077.014546:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11133 00000100:00100000:1.0:1787758077.014555:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655052032:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758077.014556:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758077.014572:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758077.014580:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655052160:192.168.201.107@tcp:38:kworker.0 00000008:00100000:0.0:1787758077.014584:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758077.014606:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588655052800/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758077.014620:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787758077.014626:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758077.014629:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758077.014635:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758077.014643:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11133 00000100:00080000:0.0:1787758077.014650:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758077.014656:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758077/real 1787758077] req@ffff8c05a9f77100 x1874588655052160/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758077.014659:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758077.014682:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758077.014683:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588655052160/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758077.014703:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787758077.014708:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588655052928/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758077.014715:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758077.014730:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655052160:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758077.014752:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655052288:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758077.014782:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758077.014796:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758077/real 1787758077] req@ffff8c05a9f76d80 x1874588655052288/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758077.014819:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588655052288/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758077.014829:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758077.014832:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758077.014836:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655052288:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758077.014868:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655052416:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758077.014940:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758077.014955:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758077/real 1787758077] req@ffff8c05a9f74e00 x1874588655052416/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758077.014970:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588655052416/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758077.014981:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758077.014985:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758077.014989:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655052416:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758077.014996:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655052544:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758077.015015:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758077.015027:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758077/real 1787758077] req@ffff8c05a9f75500 x1874588655052544/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758077.015040:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588655052544/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758077.015058:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758077.015062:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758077.015066:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655052544:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758077.015071:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655052672:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758077.015089:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758077.015099:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758077/real 1787758077] req@ffff8c05a9f75880 x1874588655052672/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758077.015113:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655052672/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758077.015123:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758077.015126:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758077.015129:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655052672:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758077.015135:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655052800:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758077.015153:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758077.015164:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758077/real 1787758077] req@ffff8c05a9f74000 x1874588655052800/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758077.015178:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588655052800/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758077.015187:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758077.015190:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758077.015195:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655052800:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758077.015201:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655052928:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758077.015219:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758077.015229:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758077/real 1787758077] req@ffff8c05a9f75180 x1874588655052928/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758077.015242:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588655052928/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758077.015252:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758077.015255:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758077.015259:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655052928:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758077.141949:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758077.590190:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758078.165942:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758078.614141:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758079.190036:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758079.638127:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758080.214004:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758080.662145:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758081.238045:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758081.686299:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758082.134062:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758082.134072:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758082.134077:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758082.134086:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11138 00000100:00080000:3.0:1787758082.134097:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758082.134104:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758082.134148:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655053056/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758082.134189:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758082.134200:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758082.134202:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758082.134209:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11138 00000100:00080000:3.0:1787758082.134216:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758082.134250:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758082.134263:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758082.134267:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655053056:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787758082.134288:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655053184/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758082.134304:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758082.134308:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758082.134310:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758082.134315:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11138 00000100:00080000:3.0:1787758082.134321:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758082.134324:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758082.134330:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787758082.134343:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758082.134345:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655053312/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758082.134358:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758082.134362:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758082.134364:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758082.134368:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11138 00000100:00080000:3.0:1787758082.134370:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758082.134374:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758082.134379:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758082.134394:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758082/real 1787758082] req@ffff8c05a9f7ad80 x1874588655053056/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758082.134395:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655053440/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758082.134446:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758082.134452:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655053056/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758082.134453:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758082.134458:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758082.134468:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11138 00000100:00080000:3.0:1787758082.134494:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758082.134502:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758082.134508:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787758082.134522:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787758082.134525:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655053568/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758082.134537:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758082.134555:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655053056:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758082.134556:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758082.134572:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758082.134593:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758082.134601:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11138 00000100:00080000:3.0:1787758082.134606:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758082.134609:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655053184:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758082.134617:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758082.134628:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758082.134646:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655053696/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758082.134669:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787758082.134680:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758082.134681:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758082.134691:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758082.134702:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11138 00000100:00100000:1.0:1787758082.134710:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758082/real 1787758082] req@ffff8c05a9f7b800 x1874588655053184/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758082.134716:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758082.134735:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758082.134744:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655053184/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787758082.134750:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787758082.134766:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758082.134774:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758082.134787:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655053184:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758082.134788:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655053824/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758082.134805:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655053312:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758082.134816:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758082.134823:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758082.134826:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758082.134835:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11138 00000100:00080000:3.0:1787758082.134845:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787758082.134849:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758082.134861:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758082.134872:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758082/real 1787758082] req@ffff8c05a9f78e00 x1874588655053312/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787758082.134874:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758082.134963:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655053312/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758082.134975:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787758082.134979:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655053952/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758082.134984:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758082.135007:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655053312:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758082.135022:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655053440:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758082.135074:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758082.135099:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758082/real 1787758082] req@ffff8c05a9f79500 x1874588655053440/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758082.135121:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655053440/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758082.135133:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758082.135137:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758082.135141:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655053440:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758082.135149:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655053568:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758082.135191:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758082.135202:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758082/real 1787758082] req@ffff8c05a9f7b100 x1874588655053568/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758082.135212:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655053568/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758082.135220:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758082.135222:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758082.135225:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655053568:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758082.135265:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655053696:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758082.135285:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758082.135295:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758082/real 1787758082] req@ffff8c05a9f79180 x1874588655053696/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758082.135306:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655053696/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758082.135319:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758082.135322:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758082.135325:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655053696:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758082.135330:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655053824:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758082.135343:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758082.135350:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758082/real 1787758082] req@ffff8c05a9f79f80 x1874588655053824/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758082.135360:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655053824/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758082.135367:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758082.135370:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758082.135372:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655053824:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758082.135377:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655053952:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758082.135390:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758082.135398:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758082/real 1787758082] req@ffff8c05a9f7bb80 x1874588655053952/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758082.135407:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655053952/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758082.135414:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758082.135417:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758082.135420:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655053952:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758082.262059:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758082.710125:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758083.286037:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758083.734239:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758084.310072:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758084.758096:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758085.333965:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758085.782284:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758086.358017:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758086.806124:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758087.254088:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758087.254104:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758087.254111:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758087.254127:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11143 00000100:00080000:0.0:1787758087.254191:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758087.254209:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758087.254287:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588655054080/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758087.254334:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758087.254340:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758087.254343:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758087.254349:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11143 00000100:00080000:0.0:1787758087.254355:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758087.254361:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758087.254376:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758087.254395:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588655054208/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758087.254412:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758087.254420:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787758087.254420:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655054080:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758087.254426:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758087.254455:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11143 00000100:00080000:0.0:1787758087.254462:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758087.254468:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758087.254475:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758087.254494:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655054336/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758087.254508:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758087.254513:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758087.254515:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758087.254521:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11143 00000100:00080000:0.0:1787758087.254526:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758087.254531:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758087.254539:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787758087.254547:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758087.254562:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588655054464/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758087.254582:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758087.254588:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758087.254592:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758087.254601:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11143 00000100:00080000:0.0:1787758087.254609:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758087.254618:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758087.254629:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758087.254641:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758087/real 1787758087] req@ffff8c05a9f75180 x1874588655054080/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758087.254659:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588655054592/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758087.254679:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588655054080/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758087.254688:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758087.254705:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758087.254712:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758087.254718:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758087.254723:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11143 00000100:00080000:1.0:1787758087.254731:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787758087.254733:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758087.254742:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655054080:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758087.254749:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758087.254762:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758087.254783:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588655054720/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758087.254793:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655054208:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758087.254826:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758087.254834:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758087.254839:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758087.254850:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11143 00000800:00000100:1.0:1787758087.254861:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758087.254863:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758087.254953:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758087.254964:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758087.254981:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758087/real 1787758087] req@ffff8c05a9f74000 x1874588655054208/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758087.254991:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588655054848/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758087.255027:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758087.255034:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787758087.255034:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588655054208/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758087.255042:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758087.255055:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11143 00000100:00080000:0.0:1787758087.255067:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758087.255081:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:1.0:1787758087.255084:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787758087.255088:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787758087.255106:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758087.255112:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655054976/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758087.255125:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655054208:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758087.255173:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655054336:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758087.255227:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758087.255249:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758087/real 1787758087] req@ffff8c05a9f75880 x1874588655054336/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758087.255271:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655054336/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758087.255283:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758087.255287:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758087.255291:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655054336:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758087.255299:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655054464:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758087.255322:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758087.255334:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758087/real 1787758087] req@ffff8c05a9f75500 x1874588655054464/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758087.255348:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588655054464/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758087.255359:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758087.255363:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758087.255367:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655054464:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758087.255374:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655054592:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758087.255394:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758087.255405:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758087/real 1787758087] req@ffff8c05a9f74e00 x1874588655054592/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758087.255428:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588655054592/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758087.255439:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758087.255442:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758087.255445:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655054592:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758087.255470:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655054720:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758087.255490:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758087.255502:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758087/real 1787758087] req@ffff8c05a9f76d80 x1874588655054720/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758087.255516:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588655054720/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758087.255527:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758087.255530:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758087.255534:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655054720:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758087.255540:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655054848:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758087.255558:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758087.255569:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758087/real 1787758087] req@ffff8c05a9f77100 x1874588655054848/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758087.255583:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588655054848/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758087.255601:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758087.255605:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758087.255609:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655054848:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758087.255615:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655054976:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758087.255632:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758087.255643:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758087/real 1787758087] req@ffff8c05a9f74a80 x1874588655054976/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758087.255657:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655054976/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758087.255667:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758087.255671:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758087.255674:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655054976:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758087.382007:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758087.830303:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758088.406025:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758088.854108:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758089.429987:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758089.878188:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758090.453990:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758090.902137:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758091.478046:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758091.926227:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758092.374006:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758092.374023:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758092.374030:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758092.374046:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11148 00000100:00080000:0.0:1787758092.374088:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758092.374102:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758092.374152:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655055104/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758092.374199:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758092.374204:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758092.374206:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758092.374212:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11148 00000100:00080000:0.0:1787758092.374217:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758092.374222:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758092.374233:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758092.374248:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655055232/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758092.374263:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758092.374268:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758092.374271:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758092.374275:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11148 00000100:00080000:0.0:1787758092.374280:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758092.374284:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758092.374291:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758092.374296:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655055104:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787758092.374309:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588655055360/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758092.374324:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758092.374329:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758092.374331:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758092.374336:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11148 00000100:00080000:0.0:1787758092.374341:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758092.374345:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758092.374351:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758092.374372:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588655055488/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758092.374391:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758092.374398:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758092.374401:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758092.374408:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11148 00000100:00080000:0.0:1787758092.374416:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758092.374424:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758092.374433:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758092.374454:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588655055616/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787758092.374470:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758092.374483:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758092.374489:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758092.374493:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758092.374503:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11148 00000100:00080000:0.0:1787758092.374512:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758092.374520:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758092.374531:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758092.374554:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758092/real 1787758092] req@ffff8c05a9f74380 x1874588655055104/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758092.374559:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655055744/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758092.374611:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655055104/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758092.374615:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758092.374633:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758092.374640:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758092.374655:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11148 00000100:00080000:0.0:1787758092.374668:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758092.374674:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758092.374676:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:1.0:1787758092.374690:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000008:00100000:0.0:1787758092.374691:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758092.374702:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655055104:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787758092.374718:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655055872/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758092.374722:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655055232:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758092.374747:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758092.374752:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758092.374755:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758092.374760:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11148 00000100:00080000:0.0:1787758092.374765:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758092.374769:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758092.374775:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758092.374797:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655056000/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787758092.374799:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758092.374842:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758092/real 1787758092] req@ffff8c05a9f76300 x1874588655055232/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758092.374862:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655055232/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758092.374875:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758092.374939:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758092.374944:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655055232:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758092.374979:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655055360:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758092.375012:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758092.375029:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758092/real 1787758092] req@ffff8c05a9f77b80 x1874588655055360/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758092.375080:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588655055360/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758092.375098:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758092.375106:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758092.375113:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655055360:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758092.375127:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655055488:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758092.375169:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758092.375186:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758092/real 1787758092] req@ffff8c05a9f76a00 x1874588655055488/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758092.375202:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588655055488/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758092.375213:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758092.375217:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758092.375221:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655055488:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758092.375229:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655055616:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758092.375251:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758092.375263:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758092/real 1787758092] req@ffff8c05a9f75c00 x1874588655055616/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758092.375277:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588655055616/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758092.375287:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758092.375290:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758092.375294:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655055616:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758092.375301:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655055744:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758092.375320:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758092.375338:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758092/real 1787758092] req@ffff8c05a9f77800 x1874588655055744/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758092.375362:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655055744/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758092.375379:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758092.375385:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758092.375393:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655055744:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758092.375405:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655055872:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758092.375442:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758092.375459:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758092/real 1787758092] req@ffff8c05a9f76680 x1874588655055872/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758092.375474:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655055872/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758092.375484:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758092.375488:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758092.375492:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655055872:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758092.375499:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655056000:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758092.375520:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758092.375531:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758092/real 1787758092] req@ffff8c05a9f74a80 x1874588655056000/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758092.375544:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655056000/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758092.375562:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758092.375566:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758092.375570:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655056000:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758092.501999:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758092.950144:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758093.526020:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758093.974072:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758094.549984:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758094.998069:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758095.574091:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758096.022240:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758096.598059:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758097.046143:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758097.494005:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758097.494019:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758097.494024:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758097.494037:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11153 00000100:00080000:0.0:1787758097.494053:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758097.494063:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758097.494113:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588655056128/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758097.494155:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758097.494160:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758097.494162:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758097.494168:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11153 00000100:00080000:0.0:1787758097.494173:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758097.494177:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758097.494189:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758097.494212:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588655056256/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758097.494228:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758097.494233:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758097.494235:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758097.494243:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11153 00000100:00080000:0.0:1787758097.494253:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758097.494263:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655056128:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758097.494264:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758097.494280:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758097.494308:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588655056384/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758097.494325:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758097.494331:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758097.494334:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758097.494340:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11153 00000100:00080000:0.0:1787758097.494356:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758097.494362:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758097.494370:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787758097.494373:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758097.494395:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588655056512/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758097.494410:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758097.494415:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758097.494419:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758097.494425:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11153 00000100:00080000:0.0:1787758097.494431:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758097.494438:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758097.494451:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758097.494479:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758097/real 1787758097] req@ffff8c05a9f77100 x1874588655056128/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758097.494491:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655056640/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758097.494542:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758097.494552:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758097.494560:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758097.494562:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588655056128/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758097.494575:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11153 00000100:00080000:0.0:1787758097.494589:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758097.494599:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758097.494601:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:1.0:1787758097.494614:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000008:00100000:0.0:1787758097.494617:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758097.494625:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655056128:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758097.494641:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655056256:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758097.494652:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588655056768/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787758097.494691:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758097.494705:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758097.494715:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758097.494720:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758097.494729:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758097/real 1787758097] req@ffff8c05a9f76d80 x1874588655056256/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758097.494738:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11153 00000100:00080000:0.0:1787758097.494755:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758097.494765:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588655056256/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758097.494767:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758097.494782:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787758097.494792:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758097.494802:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758097.494813:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588655056896/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758097.494814:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655056256:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758097.494841:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758097.494846:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758097.494849:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758097.494856:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11153 00000100:00080000:0.0:1787758097.494862:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758097.494870:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758097.494872:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655056384:192.168.201.107@tcp:38:kworker.0 00000008:00100000:0.0:1787758097.494931:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758097.494955:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588655057024/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787758097.495016:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758097.495059:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758097/real 1787758097] req@ffff8c05a9f74e00 x1874588655056384/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758097.495085:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588655056384/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758097.495110:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758097.495117:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758097.495125:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655056384:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758097.495139:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655056512:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758097.495169:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758097.495187:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758097/real 1787758097] req@ffff8c05a9f75500 x1874588655056512/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758097.495210:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588655056512/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758097.495224:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758097.495229:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758097.495233:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655056512:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758097.495240:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655056640:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758097.495261:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758097.495272:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758097/real 1787758097] req@ffff8c05a9f75880 x1874588655056640/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758097.495287:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655056640/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758097.495297:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758097.495300:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758097.495304:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655056640:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758097.495310:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655056768:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758097.495328:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758097.495339:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758097/real 1787758097] req@ffff8c05a9f74000 x1874588655056768/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758097.495352:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588655056768/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758097.495362:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758097.495366:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758097.495369:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655056768:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758097.495376:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655056896:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758097.495393:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758097.495403:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758097/real 1787758097] req@ffff8c05a9f75180 x1874588655056896/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758097.495426:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588655056896/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758097.495440:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758097.495446:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758097.495453:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655056896:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758097.495490:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655057024:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758097.495531:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758097.495550:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758097/real 1787758097] req@ffff8c05a9f76d80 x1874588655057024/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758097.495566:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588655057024/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758097.495577:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758097.495581:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758097.495585:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655057024:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758097.622045:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758098.072073:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758098.646004:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758099.094110:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758099.670037:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758100.118108:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758100.693961:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758101.142110:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758101.718104:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758102.166185:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758102.614099:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758102.614115:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758102.614120:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758102.614133:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11159 00000100:00080000:0.0:1787758102.614146:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758102.614155:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758102.614198:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588655057152/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758102.614234:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758102.614239:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758102.614242:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758102.614247:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11159 00000100:00080000:0.0:1787758102.614251:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758102.614256:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758102.614268:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758102.614300:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655057280/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758102.614312:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655057152:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758102.614324:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758102.614335:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758102.614338:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758102.614347:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11159 00000100:00080000:0.0:1787758102.614354:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758102.614363:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758102.614374:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758102.614401:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655057408/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787758102.614411:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758102.614437:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758102.614442:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758102.614445:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758102.614454:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11159 00000100:00080000:0.0:1787758102.614462:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758102.614470:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758102/real 1787758102] req@ffff8c05a9f77100 x1874588655057152/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758102.614474:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758102.614491:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758102.614512:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588655057152/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758102.614533:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787758102.614535:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655057536/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758102.614543:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758102.614552:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655057152:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758102.614577:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758102.614583:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758102.614586:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758102.614592:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11159 00000100:00080000:0.0:1787758102.614596:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758102.614601:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758102.614607:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758102.614618:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588655057664/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758102.614637:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758102.614648:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655057280:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758102.614649:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758102.614658:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758102.614672:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11159 00000100:00080000:0.0:1787758102.614682:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758102.614692:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:1.0:1787758102.614702:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:0.0:1787758102.614711:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758102.614726:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758102/real 1787758102] req@ffff8c05a9f74a80 x1874588655057280/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758102.614743:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588655057792/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758102.614751:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655057280/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758102.614786:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758102.614790:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758102.614802:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758102.614803:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787758102.614813:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758102.614814:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655057280:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758102.614835:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655057408:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758102.614860:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758102.614935:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11159 00000100:00080000:0.0:1787758102.614946:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758102.614953:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758102.614964:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758102.614976:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758102/real 1787758102] req@ffff8c05a9f76680 x1874588655057408/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758102.614990:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588655057920/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758102.615020:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758102.615028:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787758102.615029:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655057408/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758102.615032:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758102.615048:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11159 00000100:00080000:0.0:1787758102.615060:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758102.615064:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758102.615081:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:1.0:1787758102.615084:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000008:00100000:0.0:1787758102.615093:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758102.615095:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655057408:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758102.615113:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655058048/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758102.615134:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655057536:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758102.615179:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758102.615195:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758102/real 1787758102] req@ffff8c05a9f77800 x1874588655057536/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758102.615210:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655057536/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758102.615221:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758102.615225:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758102.615230:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655057536:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758102.615255:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655057664:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758102.615293:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758102.615316:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758102/real 1787758102] req@ffff8c05a9f75c00 x1874588655057664/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758102.615341:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588655057664/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758102.615355:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758102.615359:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758102.615363:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655057664:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758102.615371:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655057792:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758102.615400:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758102.615414:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758102/real 1787758102] req@ffff8c05a9f76a00 x1874588655057792/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758102.615429:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588655057792/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758102.615439:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758102.615443:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758102.615447:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655057792:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758102.615454:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655057920:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758102.615473:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758102.615484:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758102/real 1787758102] req@ffff8c05a9f77b80 x1874588655057920/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758102.615499:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588655057920/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758102.615508:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758102.615512:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758102.615515:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655057920:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758102.615522:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655058048:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758102.615540:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758102.615550:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758102/real 1787758102] req@ffff8c05a9f76300 x1874588655058048/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758102.615564:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655058048/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758102.615582:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758102.615586:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758102.615589:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655058048:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758102.742028:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758103.190092:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758103.765992:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758104.214134:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758104.789992:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758105.238293:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758105.813982:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758106.262255:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758106.838082:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758107.286172:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758107.734045:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758107.734058:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787758107.734063:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758107.734076:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11164 00000100:00080000:2.0:1787758107.734089:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758107.734098:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758107.734214:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588655058176/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758107.734265:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758107.734271:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758107.734273:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758107.734278:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11164 00000100:00080000:2.0:1787758107.734283:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758107.734287:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758107.734298:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758107.734317:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588655058304/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758107.734332:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758107.734338:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758107.734342:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758107.734350:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655058176:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787758107.734351:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11164 00000100:00080000:2.0:1787758107.734363:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758107.734373:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758107.734378:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758107.734395:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588655058432/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758107.734409:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758107.734414:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758107.734417:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758107.734423:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11164 00000100:00080000:2.0:1787758107.734427:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758107.734431:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758107.734437:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758107.734449:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588655058560/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758107.734460:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758107.734464:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758107.734467:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758107.734471:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11164 00000100:00080000:2.0:1787758107.734477:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787758107.734490:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787758107.734492:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758107.734506:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758107.734517:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588655058688/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758107.734528:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758107.734532:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758107.734534:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758107.734537:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11164 00000100:00080000:2.0:1787758107.734541:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758107.734544:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758107.734548:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758107.734557:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588655058816/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758107.734567:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758107.734570:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787758107.734572:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758107.734576:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11164 00000100:00080000:2.0:1787758107.734579:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758107.734582:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758107.734586:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758107.734596:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588655058944/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758107.734608:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758107.734613:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758107.734616:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758107.734622:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11164 00000100:00080000:2.0:1787758107.734628:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758107.734632:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758107.734637:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758107.734650:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588655059072/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758107.734686:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758107/real 1787758107] req@ffff8c05b2e70a80 x1874588655058176/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758107.734720:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588655058176/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758107.734734:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758107.734740:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758107.734745:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655058176:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758107.734806:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655058304:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758107.734935:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758107.734956:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758107/real 1787758107] req@ffff8c05b2e71180 x1874588655058304/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758107.734973:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588655058304/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758107.734993:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758107.734998:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758107.735002:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655058304:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758107.735011:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655058432:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758107.735038:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758107.735060:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758107/real 1787758107] req@ffff8c05b2e72680 x1874588655058432/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758107.735074:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588655058432/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758107.735084:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758107.735088:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758107.735092:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655058432:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758107.735099:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655058560:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758107.735119:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758107.735130:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758107/real 1787758107] req@ffff8c05b2e72d80 x1874588655058560/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758107.735144:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588655058560/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758107.735154:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758107.735157:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758107.735162:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655058560:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758107.735169:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655058688:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758107.735188:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758107.735199:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758107/real 1787758107] req@ffff8c05b2e73100 x1874588655058688/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758107.735213:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588655058688/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758107.735223:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758107.735226:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758107.735230:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655058688:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758107.735236:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655058816:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758107.735253:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758107.735264:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758107/real 1787758107] req@ffff8c05b2e72a00 x1874588655058816/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758107.735288:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588655058816/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758107.735298:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758107.735301:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758107.735305:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655058816:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758107.735310:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655058944:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758107.735328:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758107.735339:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758107/real 1787758107] req@ffff8c05b2e72300 x1874588655058944/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758107.735353:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588655058944/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758107.735363:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758107.735366:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758107.735369:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655058944:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758107.735375:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655059072:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758107.735393:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758107.735403:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758107/real 1787758107] req@ffff8c05b2e70000 x1874588655059072/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758107.735415:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588655059072/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758107.735425:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758107.735445:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758107.735449:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655059072:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758107.862010:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758108.310100:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758108.885997:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758109.334120:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758109.909976:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758110.358254:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758110.934025:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758111.382035:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758111.957986:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758112.406124:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758112.854041:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758112.854055:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758112.854060:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758112.854073:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11169 00000100:00080000:3.0:1787758112.854087:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758112.854096:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758112.854148:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655059200/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758112.854200:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758112.854208:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758112.854212:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758112.854221:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11169 00000100:00080000:3.0:1787758112.854229:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758112.854238:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758112.854255:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758112.854281:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655059200:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787758112.854283:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655059328/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758112.854310:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758112.854315:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758112.854318:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758112.854323:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11169 00000100:00080000:3.0:1787758112.854328:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758112.854333:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758112.854341:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758112.854359:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655059456/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758112.854372:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758112.854377:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758112.854380:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758112.854384:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11169 00000100:00080000:3.0:1787758112.854389:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787758112.854398:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758112.854399:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758112.854413:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758112.854431:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655059584/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758112.854443:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758112.854448:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758112.854451:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758112.854458:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11169 00000100:00080000:3.0:1787758112.854464:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758112.854472:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758112/real 1787758112] req@ffff8c05a9f7bb80 x1874588655059200/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758112.854474:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758112.854488:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758112.854511:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655059712/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758112.854526:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758112.854532:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655059200/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758112.854534:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758112.854540:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758112.854556:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11169 00000100:00080000:3.0:1787758112.854567:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758112.854568:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758112.854574:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:1.0:1787758112.854583:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000008:00100000:3.0:1787758112.854587:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758112.854594:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655059200:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787758112.854609:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655059840/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758112.854622:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758112.854626:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758112.854628:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758112.854633:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11169 00000100:00080000:3.0:1787758112.854636:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758112.854640:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758112.854647:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758112.854648:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655059328:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758112.854665:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655059968/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758112.854681:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758112.854717:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758112.854723:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787758112.854732:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758112.854735:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11169 00000100:00080000:3.0:1787758112.854749:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758112.854759:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758112.854765:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758112/real 1787758112] req@ffff8c05a9f79f80 x1874588655059328/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787758112.854775:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758112.854790:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655059328/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758112.854810:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655060096/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758112.854832:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758112.854848:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758112.854862:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655059328:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758112.854927:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655059456:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758112.854980:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758112.854997:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758112/real 1787758112] req@ffff8c05a9f79180 x1874588655059456/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758112.855013:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655059456/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758112.855031:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758112.855035:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758112.855040:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655059456:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758112.855047:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655059584:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758112.855069:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758112.855080:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758112/real 1787758112] req@ffff8c05a9f7b100 x1874588655059584/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758112.855095:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655059584/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758112.855111:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758112.855118:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758112.855124:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655059584:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758112.855136:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655059712:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758112.855176:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758112.855199:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758112/real 1787758112] req@ffff8c05a9f79500 x1874588655059712/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758112.855223:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655059712/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758112.855234:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758112.855238:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758112.855243:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655059712:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758112.855251:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655059840:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758112.855278:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758112.855291:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758112/real 1787758112] req@ffff8c05a9f78e00 x1874588655059840/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758112.855305:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655059840/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758112.855315:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758112.855319:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758112.855322:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655059840:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758112.855348:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655059968:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758112.855368:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758112.855380:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758112/real 1787758112] req@ffff8c05a9f7b800 x1874588655059968/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758112.855394:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655059968/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758112.855404:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758112.855407:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758112.855411:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655059968:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758112.855417:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655060096:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758112.855435:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758112.855446:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758112/real 1787758112] req@ffff8c05a9f7ad80 x1874588655060096/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758112.855460:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655060096/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758112.855470:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758112.855473:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758112.855477:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655060096:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758112.982029:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758113.430087:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758114.006011:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758114.454180:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758115.030021:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758115.478153:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758116.053991:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758116.502188:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758117.078066:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758117.526211:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758117.974006:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758117.974019:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758117.974025:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758117.974038:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11174 00000100:00080000:3.0:1787758117.974055:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758117.974068:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758117.974127:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655060224/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758117.974177:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758117.974185:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758117.974189:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758117.974197:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11174 00000100:00080000:3.0:1787758117.974204:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758117.974210:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758117.974226:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758117.974255:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655060352/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758117.974269:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655060224:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758117.974291:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758117.974297:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758117.974300:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758117.974308:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11174 00000100:00080000:3.0:1787758117.974315:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758117.974322:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758117.974332:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758117.974352:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655060480/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758117.974367:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758117.974372:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758117.974374:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758117.974382:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11174 00000800:00000100:1.0:1787758117.974385:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758117.974390:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758117.974397:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758117.974405:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758117.974421:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655060608/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758117.974434:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758117.974439:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758117.974442:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758117.974451:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11174 00000100:00100000:1.0:1787758117.974452:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758117/real 1787758117] req@ffff8c05a9f7b480 x1874588655060224/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758117.974456:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758117.974468:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758117.974481:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758117.974502:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655060224/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758117.974506:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655060736/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758117.974549:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758117.974568:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787758117.974573:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758117.974582:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787758117.974584:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655060224:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758117.974586:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758117.974630:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11174 00000100:00080000:3.0:1787758117.974637:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758117.974642:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758117.974648:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758117.974672:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655060352:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758117.974672:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655060864/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758117.974701:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758117.974705:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758117.974708:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758117.974713:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11174 00000100:00080000:3.0:1787758117.974719:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758117.974724:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758117.974730:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787758117.974744:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758117.974750:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655060992/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758117.974767:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758117.974771:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758117.974774:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758117.974784:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11174 00000100:00100000:1.0:1787758117.974788:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758117/real 1787758117] req@ffff8c05a9f78380 x1874588655060352/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758117.974790:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758117.974801:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758117.974815:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758117.974836:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655061120/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758117.974850:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655060352/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758117.974867:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758117.974872:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758117.974931:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655060352:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758117.974942:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655060480:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758117.974976:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758117.974992:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758117/real 1787758117] req@ffff8c05a9f78a80 x1874588655060480/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758117.975007:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655060480/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758117.975017:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758117.975022:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758117.975026:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655060480:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758117.975033:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655060608:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758117.975054:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758117.975065:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758117/real 1787758117] req@ffff8c05a9f7a680 x1874588655060608/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758117.975079:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655060608/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758117.975097:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758117.975101:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758117.975105:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655060608:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758117.975111:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655060736:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758117.975129:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758117.975141:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758117/real 1787758117] req@ffff8c05a9f7aa00 x1874588655060736/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758117.975154:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655060736/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758117.975164:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758117.975167:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758117.975171:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655060736:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758117.975192:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655060864:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758117.975212:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758117.975224:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758117/real 1787758117] req@ffff8c05a9f7a300 x1874588655060864/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758117.975238:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655060864/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758117.975247:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758117.975251:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758117.975256:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655060864:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758117.975262:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655060992:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758117.975281:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758117.975292:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758117/real 1787758117] req@ffff8c05a9f79880 x1874588655060992/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758117.975305:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655060992/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758117.975315:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758117.975318:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758117.975321:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655060992:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758117.975327:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655061120:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758117.975345:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758117.975355:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758117/real 1787758117] req@ffff8c05a9f78700 x1874588655061120/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758117.975369:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655061120/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758117.975378:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758117.975382:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758117.975385:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655061120:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758118.101958:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758118.550222:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758119.126029:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758119.574145:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758120.150029:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758120.598244:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758121.174022:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758121.622094:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758122.198032:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758122.646129:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758123.093983:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758123.093994:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758123.093999:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758123.094009:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11179 00000100:00080000:1.0:1787758123.094020:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758123.094028:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758123.094068:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588655061248/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758123.094101:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758123.094106:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758123.094109:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758123.094114:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11179 00000100:00080000:1.0:1787758123.094119:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758123.094125:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758123.094133:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655061248:192.168.201.107@tcp:250:kworker.0 00000008:00100000:1.0:1787758123.094135:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758123.094156:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655061376/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758123.094169:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758123.094182:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758123.094184:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758123.094189:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11179 00000100:00080000:1.0:1787758123.094195:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758123.094200:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758123.094206:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758123.094220:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655061504/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758123.094233:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787758123.094233:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758123.094241:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758123.094243:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758123.094248:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11179 00000100:00080000:1.0:1787758123.094252:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758123.094258:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758123.094264:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758123.094278:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655061632/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758123.094289:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787758123.094295:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758123/real 1787758123] req@ffff8c05b2d9a300 x1874588655061248/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758123.094296:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758123.094300:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758123.094311:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11179 00000100:00080000:1.0:1787758123.094317:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787758123.094321:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588655061248/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758123.094326:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:3.0:1787758123.094337:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787758123.094339:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787758123.094343:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758123.094347:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655061248:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758123.094359:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655061760/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787758123.094368:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655061376:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758123.094377:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758123.094382:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758123.094385:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758123.094391:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11179 00000800:00000100:3.0:1787758123.094394:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758123.094400:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758123.094408:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758123.094409:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758123/real 1787758123] req@ffff8c05b2d98380 x1874588655061376/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787758123.094417:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758123.094422:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655061376/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758123.094435:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655061888/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758123.094444:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758123.094450:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787758123.094451:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787758123.094455:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655061376:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758123.094461:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787758123.094464:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655061504:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758123.094466:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758123.094474:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11179 00000100:00080000:1.0:1787758123.094481:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787758123.094483:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758123.094489:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758123.094494:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758123/real 1787758123] req@ffff8c05b2d9aa00 x1874588655061504/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787758123.094498:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758123.094506:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655061504/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758123.094532:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655062016/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758123.094534:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758123.094544:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758123.094551:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655061504:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758123.094555:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758123.094561:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787758123.094562:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655061632:192.168.201.107@tcp:8:kworker.0 00000100:00080000:1.0:1787758123.094565:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758123.094574:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11179 00000100:00080000:1.0:1787758123.094579:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758123.094585:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:3.0:1787758123.094589:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:1.0:1787758123.094591:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758123.094600:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758123/real 1787758123] req@ffff8c05b2d9bb80 x1874588655061632/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758123.094610:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655062144/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787758123.094619:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655061632/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758123.094634:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758123.094638:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758123.094643:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655061632:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758123.094661:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655061760:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758123.094678:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758123.094685:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758123/real 1787758123] req@ffff8c05b2d98700 x1874588655061760/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758123.094693:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655061760/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758123.094699:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758123.094701:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758123.094703:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655061760:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758123.094706:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655061888:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758123.094716:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758123.094722:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758123/real 1787758123] req@ffff8c05b2d9b100 x1874588655061888/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758123.094729:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655061888/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758123.094735:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758123.094738:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758123.094740:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655061888:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758123.094744:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655062016:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758123.094754:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758123.094762:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758123/real 1787758123] req@ffff8c05b2d99880 x1874588655062016/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758123.094770:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655062016/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758123.094775:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758123.094777:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758123.094778:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655062016:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758123.094782:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655062144:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758123.094793:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758123.094807:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758123/real 1787758123] req@ffff8c05b2d99f80 x1874588655062144/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758123.094815:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655062144/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758123.094820:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758123.094822:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758123.094824:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655062144:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758123.221962:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758123.670099:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758124.245963:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758124.694230:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758125.269972:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758125.718136:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758126.294025:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758126.742118:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758127.317963:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758127.766004:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758128.214057:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758128.214074:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758128.214081:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758128.214114:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11184 00000100:00080000:3.0:1787758128.214125:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758128.214134:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758128.214193:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655062272/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758128.214244:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758128.214250:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758128.214252:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758128.214259:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11184 00000100:00080000:3.0:1787758128.214266:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758128.214273:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758128.214287:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758128.214311:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655062272:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787758128.214318:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655062400/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758128.214341:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758128.214345:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758128.214348:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758128.214354:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11184 00000100:00080000:3.0:1787758128.214358:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758128.214363:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758128.214370:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758128.214387:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655062528/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758128.214407:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758128.214414:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758128.214438:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758128.214457:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11184 00000100:00080000:3.0:1787758128.214466:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758128.214475:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:1.0:1787758128.214487:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:3.0:1787758128.214492:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758128.214523:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655062656/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758128.214544:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758128.214551:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758128.214555:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758128.214567:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758128/real 1787758128] req@ffff8c05a9f78700 x1874588655062272/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758128.214568:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11184 00000100:00080000:3.0:1787758128.214583:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758128.214600:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758128.214611:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655062272/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787758128.214613:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787758128.214638:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758128.214648:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758128.214650:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655062784/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758128.214658:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655062272:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758128.214684:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758128.214689:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758128.214692:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758128.214700:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11184 00000100:00100000:1.0:1787758128.214707:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655062400:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758128.214708:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758128.214736:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758128.214746:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787758128.214755:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758128.214771:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655062912/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758128.214781:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758128/real 1787758128] req@ffff8c05a9f79880 x1874588655062400/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758128.214801:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758128.214809:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758128.214812:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758128.214816:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655062400/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758128.214831:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11184 00000100:00080000:1.0:1787758128.214835:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758128.214842:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758128.214846:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787758128.214855:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758128.214858:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655062400:192.168.201.107@tcp:38:kworker.0 00000008:00100000:3.0:1787758128.214867:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758128.214928:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655062528:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758128.214977:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655063040/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787758128.214991:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758128.215005:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758128.215011:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758128.215014:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758128.215023:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11184 00000100:00100000:1.0:1787758128.215028:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758128/real 1787758128] req@ffff8c05a9f7a300 x1874588655062528/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758128.215032:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758128.215051:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758128.215060:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655062528/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787758128.215066:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787758128.215084:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758128.215091:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758128.215095:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655062528:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758128.215107:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655063168/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758128.215109:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655062656:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758128.215165:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758128.215196:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758128/real 1787758128] req@ffff8c05a9f7aa00 x1874588655062656/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758128.215222:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655062656/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758128.215239:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758128.215245:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758128.215252:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655062656:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758128.215263:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655062784:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758128.215305:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758128.215329:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758128/real 1787758128] req@ffff8c05a9f7a680 x1874588655062784/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758128.215351:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655062784/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758128.215363:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758128.215366:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758128.215370:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655062784:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758128.215394:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655062912:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758128.215458:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758128.215483:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758128/real 1787758128] req@ffff8c05a9f78a80 x1874588655062912/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758128.215508:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655062912/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758128.215526:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758128.215531:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758128.215535:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655062912:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758128.215544:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655063040:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758128.215572:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758128.215587:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758128/real 1787758128] req@ffff8c05a9f78380 x1874588655063040/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758128.215601:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655063040/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758128.215620:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758128.215624:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758128.215628:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655063040:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758128.215634:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655063168:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758128.215652:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758128.215664:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758128/real 1787758128] req@ffff8c05a9f7b480 x1874588655063168/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758128.215677:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655063168/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758128.215687:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758128.215690:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758128.215694:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655063168:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758128.342030:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758128.790197:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758129.366006:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758129.814163:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758130.390092:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758130.838223:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758131.413966:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758131.862018:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758132.437982:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758132.886233:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758133.333965:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758133.333976:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758133.333981:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758133.333991:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11189 00000100:00080000:3.0:1787758133.334003:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758133.334012:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758133.334060:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655063296/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758133.334089:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758133.334095:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758133.334099:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758133.334105:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11189 00000100:00080000:3.0:1787758133.334111:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758133.334117:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758133.334130:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758133.334154:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655063424/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758133.334168:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655063296:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758133.334171:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758133.334177:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758133.334180:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758133.334186:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11189 00000100:00080000:3.0:1787758133.334193:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758133.334199:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758133.334206:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758133.334222:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655063552/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758133.334246:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758133.334252:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758133.334255:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758133.334262:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11189 00000100:00080000:3.0:1787758133.334268:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758133.334274:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758133.334282:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758133.334302:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655063680/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758133.334303:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655063424:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758133.334321:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758133.334327:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758133.334349:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758133.334357:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11189 00000100:00100000:1.0:1787758133.334381:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655063552:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758133.334428:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655063680:192.168.201.107@tcp:8:kworker.0 00000100:00080000:3.0:1787758133.334486:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758133.334496:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758133.335967:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758133.336116:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655063808/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758133.336233:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758133.336246:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758133.336250:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758133.336258:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11189 00000100:00080000:3.0:1787758133.336267:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758133.336274:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758133.336283:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758133.336304:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655063936/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758133.336319:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758133.336324:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758133.336328:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758133.336355:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11189 00000100:00080000:3.0:1787758133.336359:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758133.336363:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758133.336370:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758133.336387:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655064064/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758133.336402:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758133.336407:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758133.336410:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758133.336415:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11189 00000100:00080000:3.0:1787758133.336420:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758133.336426:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758133.336432:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758133.336446:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655064192/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758133.336522:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655063808:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758133.336591:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655063936:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758133.336626:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655064064:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758133.336657:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655064192:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758133.462002:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758133.909998:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758134.486044:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758134.934213:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758135.510011:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758135.958128:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787758136.406310:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.107:988 00000400:00000100:0.0:1787758136.406371:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.107@tcp at host 192.168.201.107 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787758136.406414:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787758136.406480:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787758136.406498:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787758136.406513:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787758136.406533:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:1.0:1787758136.406569:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758133/real 1787758136] req@ffff8c05a9f79500 x1874588655063680/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787758136.406587:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787758136.406616:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:1.0:1787758136.406638:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655063680/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787758136.406642:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00080000:1.0:1787758136.406675:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758136.406714:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758136.406731:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655063680:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758136.406748:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758133/real 1787758136] req@ffff8c05a9f7b800 x1874588655063424/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758136.406776:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655063424/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758136.406793:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758136.406799:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758136.406806:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655063424:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758136.406820:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758133/real 1787758136] req@ffff8c05a9f78e00 x1874588655063552/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758136.406844:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655063552/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758136.406858:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758136.406865:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758136.406871:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655063552:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758136.406939:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758133/real 1787758136] req@ffff8c05a9f7ad80 x1874588655063296/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758136.406967:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655063296/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758136.406985:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758136.406991:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758136.407010:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655063296:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758136.407055:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758133/real 1787758136] req@ffff8c05a9f7b100 x1874588655063808/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758136.407081:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655063808/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758136.407097:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758136.407102:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758136.407108:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655063808:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758136.407120:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758133/real 1787758136] req@ffff8c05a9f79180 x1874588655063936/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758136.407156:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655063936/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758136.407172:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758136.407179:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758136.407186:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655063936:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758136.407200:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758133/real 1787758136] req@ffff8c05a9f79f80 x1874588655064064/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758136.407224:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655064064/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758136.407240:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758136.407311:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758136.407319:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655064064:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758136.407332:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758133/real 1787758136] req@ffff8c05a9f7bb80 x1874588655064192/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758136.407359:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655064192/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758136.407376:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758136.407382:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758136.407389:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655064192:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758136.534048:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758136.982209:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758137.558010:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758138.006140:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758138.454039:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758138.454049:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758138.454053:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758138.454062:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11194 00000100:00080000:0.0:1787758138.454073:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758138.454080:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758138.454113:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655064320/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758138.454139:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758138.454142:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758138.454144:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758138.454148:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11194 00000100:00080000:0.0:1787758138.454150:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758138.454162:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758138.454170:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758138.454182:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655064448/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758138.454191:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758138.454194:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758138.454196:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758138.454199:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11194 00000100:00080000:0.0:1787758138.454203:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758138.454206:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758138.454211:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758138.454219:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588655064576/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758138.454231:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655064320:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758138.454237:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758138.454257:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758138.454260:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758138.454269:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11194 00000100:00080000:0.0:1787758138.454274:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758138.454280:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758138.454287:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758138.454306:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588655064704/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758138.454321:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758138.454326:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758138.454328:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758138.454333:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11194 00000100:00080000:0.0:1787758138.454340:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787758138.454347:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758138.454353:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758138.454362:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758138.454381:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588655064832/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758138.454394:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758138.454398:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758138.454401:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758138.454408:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758138/real 1787758138] req@ffff8c05a9f74380 x1874588655064320/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758138.454414:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11194 00000100:00080000:0.0:1787758138.454425:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758138.454434:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758138.454441:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758138.454444:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655064320/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758138.454467:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758138.454472:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758138.454478:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655064320:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758138.454488:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655064448:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758138.454521:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655064960/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787758138.454528:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758138.454550:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758138/real 1787758138] req@ffff8c05a9f76300 x1874588655064448/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758138.454579:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758138.454586:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655064448/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758138.454588:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758138.454593:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758138.454607:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11194 00000100:00080000:1.0:1787758138.454612:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758138.454616:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758138.454623:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787758138.454629:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758138.454634:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655064448:192.168.201.107@tcp:38:kworker.0 00000008:00100000:0.0:1787758138.454640:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758138.454664:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655065088/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758138.454676:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655064576:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758138.454688:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758138.454695:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758138.454698:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758138.454706:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11194 00000100:00080000:0.0:1787758138.454713:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787758138.454717:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758138.454724:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758138.454734:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758138.454741:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758138/real 1787758138] req@ffff8c05a9f77b80 x1874588655064576/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758138.454759:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655065216/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758138.454765:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588655064576/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758138.454798:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758138.454806:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758138.454812:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655064576:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758138.454823:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655064704:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758138.454853:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758138.454868:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758138/real 1787758138] req@ffff8c05a9f76a00 x1874588655064704/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758138.454919:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588655064704/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758138.454931:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758138.454936:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758138.454940:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655064704:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758138.454948:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655064832:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758138.454979:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758138.454997:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758138/real 1787758138] req@ffff8c05a9f75c00 x1874588655064832/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758138.455014:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588655064832/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758138.455026:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758138.455031:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758138.455037:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655064832:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758138.455047:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655064960:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758138.455074:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758138.455089:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758138/real 1787758138] req@ffff8c05a9f77800 x1874588655064960/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758138.455108:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655064960/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758138.455119:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758138.455125:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758138.455130:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655064960:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758138.455154:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655065088:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758138.455183:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758138.455198:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758138/real 1787758138] req@ffff8c05a9f76680 x1874588655065088/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758138.455213:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655065088/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758138.455223:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758138.455227:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758138.455231:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655065088:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758138.455257:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655065216:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758138.455285:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758138.455301:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758138/real 1787758138] req@ffff8c05a9f74a80 x1874588655065216/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758138.455316:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655065216/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758138.455336:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758138.455340:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758138.455345:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655065216:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758138.581935:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758139.030084:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758139.605956:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758140.054027:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758140.629986:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758141.078190:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758141.653987:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758142.102444:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758142.678010:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758143.126253:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758143.574070:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758143.574082:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758143.574086:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758143.574098:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11199 00000100:00080000:1.0:1787758143.574111:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758143.574119:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758143.574208:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655065344/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758143.574258:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758143.574265:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758143.574269:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758143.574276:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11199 00000100:00080000:1.0:1787758143.574284:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758143.574303:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758143.574315:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758143.574333:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655065344:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758143.574339:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655065472/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758143.574358:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758143.574364:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758143.574366:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758143.574371:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11199 00000100:00080000:1.0:1787758143.574376:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758143.574380:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758143.574386:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758143.574401:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655065600/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758143.574413:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758143.574418:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758143.574424:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787758143.574425:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758143.574432:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11199 00000100:00080000:1.0:1787758143.574439:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758143.574443:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758143.574450:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758143.574463:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655065728/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758143.574473:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758143.574477:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758143.574479:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758143.574482:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11199 00000100:00080000:1.0:1787758143.574497:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758143.574502:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758143.574509:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758143.574527:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655065856/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758143.574530:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758143/real 1787758143] req@ffff8c05b2d9b480 x1874588655065344/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758143.574563:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758143.574579:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758143.574584:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787758143.574591:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655065344/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758143.574594:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11199 00000100:00080000:1.0:1787758143.574608:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758143.574615:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758143.574622:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787758143.574627:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758143.574638:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655065344:192.168.201.107@tcp:250:kworker.0 00000008:00100000:1.0:1787758143.574640:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758143.574679:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655065984/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758143.574685:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655065472:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758143.574712:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758143.574717:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758143.574721:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787758143.574730:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758143.574744:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11199 00000100:00080000:1.0:1787758143.574752:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787758143.574759:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758143/real 1787758143] req@ffff8c05b2d98000 x1874588655065472/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758143.574761:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758143.574774:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758143.574797:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655066112/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758143.574807:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655065472/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758143.574815:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758143.574829:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758143.574836:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758143.574850:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11199 00000100:00080000:0.0:1787758143.574855:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758143.574855:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758143.574866:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787758143.574873:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758143.574912:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655065472:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758143.574922:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655065600:192.168.201.107@tcp:38:kworker.0 00000008:00100000:1.0:1787758143.574943:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787758143.574959:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758143.574978:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655066240/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758143.574984:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758143/real 1787758143] req@ffff8c05b2d99f80 x1874588655065600/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758143.575036:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655065600/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758143.575058:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758143.575062:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758143.575067:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655065600:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758143.575074:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655065728:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758143.575094:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758143.575105:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758143/real 1787758143] req@ffff8c05b2d99880 x1874588655065728/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758143.575119:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655065728/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758143.575134:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758143.575140:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758143.575146:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655065728:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758143.575182:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655065856:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758143.575228:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758143.575248:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758143/real 1787758143] req@ffff8c05b2d9b100 x1874588655065856/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758143.575263:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655065856/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758143.575273:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758143.575277:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758143.575281:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655065856:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758143.575316:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655065984:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758143.575351:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758143.575371:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758143/real 1787758143] req@ffff8c05b2d98700 x1874588655065984/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758143.575394:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655065984/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758143.575407:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758143.575411:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758143.575415:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655065984:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758143.575422:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655066112:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758143.575447:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758143.575459:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758143/real 1787758143] req@ffff8c05b2d9bb80 x1874588655066112/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758143.575473:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655066112/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758143.575481:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758143.575484:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758143.575488:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655066112:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758143.575493:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655066240:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758143.575510:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758143.575520:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758143/real 1787758143] req@ffff8c05b2d9aa00 x1874588655066240/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758143.575533:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655066240/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758143.575542:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758143.575545:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758143.575548:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655066240:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758143.702067:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758144.150181:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758144.725953:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758145.174218:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758145.750023:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758146.198189:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758146.774016:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758147.222331:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758147.798041:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758148.246156:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758148.694118:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758148.694133:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758148.694140:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758148.694156:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11205 00000100:00080000:0.0:1787758148.694175:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758148.694186:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758148.694247:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588655066368/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758148.694296:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758148.694301:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758148.694304:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758148.694309:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11205 00000100:00080000:0.0:1787758148.694315:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758148.694320:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758148.694333:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758148.694353:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588655066496/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758148.694373:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655066368:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758148.694399:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758148.694405:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758148.694408:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758148.694415:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11205 00000100:00080000:0.0:1787758148.694420:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758148.694424:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758148.694432:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758148.694450:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588655066624/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758148.694463:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758148.694467:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758148.694469:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758148.694479:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11205 00000800:00000100:1.0:1787758148.694480:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758148.694489:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758148.694495:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758148.694501:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758148.694513:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588655066752/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758148.694525:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758148.694529:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758148.694531:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758148.694534:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11205 00000100:00080000:0.0:1787758148.694539:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758148.694544:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758148.694548:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758148.694558:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655066880/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758148.694569:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758148.694574:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758148.694576:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758148.694581:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11205 00000100:00080000:0.0:1787758148.694587:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758148.694592:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758148.694599:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758148.694612:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588655067008/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758148.694625:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758148.694629:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758148.694632:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758148.694642:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758148/real 1787758148] req@ffff8c05a9f77100 x1874588655066368/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758148.694643:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11205 00000100:00080000:0.0:1787758148.694654:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758148.694666:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758148.694679:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758148.694694:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588655066368/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758148.694702:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588655067136/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758148.694738:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758148.694748:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758148.694752:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758148.694753:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758148.694775:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787758148.694776:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11205 00000100:00080000:0.0:1787758148.694789:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758148.694795:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655066368:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758148.694802:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758148.694812:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758148.694827:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655067264/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758148.694841:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655066496:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758148.694925:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758148.694947:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758148/real 1787758148] req@ffff8c05a9f76d80 x1874588655066496/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758148.694974:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588655066496/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758148.694992:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758148.694999:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758148.695006:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655066496:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758148.695019:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655066624:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758148.695086:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758148.695112:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758148/real 1787758148] req@ffff8c05a9f75180 x1874588655066624/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758148.695139:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588655066624/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758148.695151:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758148.695156:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758148.695160:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655066624:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758148.695169:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655066752:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758148.695200:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758148.695215:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758148/real 1787758148] req@ffff8c05a9f74000 x1874588655066752/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758148.695229:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588655066752/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758148.695248:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758148.695252:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758148.695256:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655066752:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758148.695262:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655066880:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758148.695281:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758148.695293:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758148/real 1787758148] req@ffff8c05a9f75880 x1874588655066880/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758148.695306:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655066880/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758148.695316:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758148.695320:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758148.695324:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655066880:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758148.695330:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655067008:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758148.695347:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758148.695358:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758148/real 1787758148] req@ffff8c05a9f75500 x1874588655067008/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758148.695372:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588655067008/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758148.695381:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758148.695384:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758148.695389:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655067008:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758148.695396:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655067136:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758148.695414:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758148.695424:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758148/real 1787758148] req@ffff8c05a9f74e00 x1874588655067136/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758148.695438:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588655067136/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758148.695448:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758148.695451:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758148.695455:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655067136:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758148.695481:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655067264:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758148.695508:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758148.695528:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758148/real 1787758148] req@ffff8c05a9f74a80 x1874588655067264/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758148.695552:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655067264/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758148.695570:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758148.695577:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758148.695584:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655067264:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758148.822009:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758149.270252:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758149.846036:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758150.294195:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758150.870041:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758151.318249:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758151.894033:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758152.342145:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758152.917972:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758153.366240:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758153.813939:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758153.813953:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758153.813956:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758153.813964:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11210 00000100:00080000:0.0:1787758153.813985:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758153.813991:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758153.814034:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655067392/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758153.814056:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758153.814058:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758153.814060:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758153.814063:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11210 00000100:00080000:0.0:1787758153.814065:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758153.814069:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758153.814076:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758153.814086:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655067520/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758153.814093:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758153.814096:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758153.814097:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758153.814100:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11210 00000100:00080000:0.0:1787758153.814102:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758153.814106:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758153.814108:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655067392:192.168.201.107@tcp:250:kworker.0 00000008:00100000:0.0:1787758153.814109:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758153.814119:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588655067648/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758153.814127:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758153.814129:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758153.814131:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758153.814133:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11210 00000100:00080000:0.0:1787758153.814135:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758153.814138:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758153.814141:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758153.814149:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588655067776/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758153.814155:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758153.814158:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758153.814160:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758153.814162:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11210 00000100:00080000:0.0:1787758153.814171:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758153.814173:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758153.814177:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787758153.814179:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758153.814188:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588655067904/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758153.814197:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758153.814201:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758153.814203:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758153.814207:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11210 00000100:00080000:0.0:1787758153.814211:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758153.814216:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758153.814220:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758153/real 1787758153] req@ffff8c05a9f76680 x1874588655067392/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787758153.814223:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758153.814245:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655068032/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758153.814246:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655067392/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758153.814276:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758153.814277:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758153.814284:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758153.814285:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787758153.814286:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758153.814291:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655067392:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758153.814294:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11210 00000100:00080000:0.0:1787758153.814298:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758153.814301:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655067520:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758153.814305:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758153.814312:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758153.814324:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655068160/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787758153.814327:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758153.814337:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758153.814339:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758153.814341:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758153.814343:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758153/real 1787758153] req@ffff8c05a9f77800 x1874588655067520/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758153.814347:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11210 00000100:00080000:0.0:1787758153.814352:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758153.814359:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758153.814364:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758153.814364:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655067520/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758153.814373:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787758153.814375:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655068288/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758153.814378:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758153.814385:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655067520:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758153.814412:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655067648:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758153.814425:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758153.814431:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758153/real 1787758153] req@ffff8c05a9f75c00 x1874588655067648/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758153.814444:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588655067648/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758153.814449:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758153.814451:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758153.814453:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655067648:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758153.814456:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655067776:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758153.814466:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758153.814472:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758153/real 1787758153] req@ffff8c05a9f76a00 x1874588655067776/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758153.814479:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588655067776/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758153.814484:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758153.814486:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758153.814488:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655067776:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758153.814491:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655067904:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758153.814501:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758153.814507:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758153/real 1787758153] req@ffff8c05a9f77b80 x1874588655067904/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758153.814514:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588655067904/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758153.814523:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758153.814525:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758153.814527:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655067904:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758153.814530:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655068032:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758153.814539:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758153.814545:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758153/real 1787758153] req@ffff8c05a9f76300 x1874588655068032/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758153.814552:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655068032/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758153.814568:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758153.814570:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758153.814572:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655068032:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758153.814575:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655068160:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758153.814584:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758153.814590:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758153/real 1787758153] req@ffff8c05a9f74380 x1874588655068160/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758153.814596:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655068160/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758153.814601:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758153.814602:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758153.814605:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655068160:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758153.814608:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655068288:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758153.814618:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758153.814623:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758153/real 1787758153] req@ffff8c05a9f74a80 x1874588655068288/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758153.814630:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655068288/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758153.814635:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758153.814637:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758153.814639:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655068288:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758153.941984:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758154.390052:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758154.965971:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758155.414095:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758155.990001:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758156.438068:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758157.013963:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758157.462113:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758158.037987:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758158.486042:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758158.933982:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758158.933991:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758158.933994:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758158.934003:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11215 00000100:00080000:1.0:1787758158.934014:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758158.934023:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758158.934062:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655068416/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758158.934087:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758158.934090:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758158.934091:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758158.934095:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11215 00000100:00080000:1.0:1787758158.934099:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758158.934102:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758158.934111:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758158.934124:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655068416:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758158.934125:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588655068544/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758158.934140:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758158.934142:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758158.934143:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758158.934147:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11215 00000100:00080000:1.0:1787758158.934150:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758158.934154:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758158.934159:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758158.934173:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655068672/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758158.934185:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787758158.934191:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758158.934191:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758158.934195:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758158.934202:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11215 00000100:00080000:1.0:1787758158.934214:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758158.934218:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758158.934224:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758158.934230:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758158/real 1787758158] req@ffff8c05b2d98380 x1874588655068416/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758158.934236:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655068800/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758158.934251:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787758158.934255:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655068416/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758158.934256:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758158.934259:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758158.934267:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11215 00000100:00080000:1.0:1787758158.934272:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758158.934275:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758158.934276:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787758158.934282:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000008:00100000:1.0:1787758158.934282:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758158.934290:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655068416:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758158.934296:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655068928/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758158.934303:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758158.934306:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758158.934307:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758158.934310:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11215 00000100:00080000:1.0:1787758158.934312:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787758158.934318:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655068544:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758158.934318:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758158.934325:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758158.934336:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655069056/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758158.934344:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787758158.934345:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758158.934350:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758158.934353:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758158.934357:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11215 00000100:00080000:1.0:1787758158.934360:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787758158.934363:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758158/real 1787758158] req@ffff8c05b2d9a300 x1874588655068544/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758158.934366:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758158.934373:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758158.934385:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655069184/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758158.934391:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588655068544/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758158.934395:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758158.934404:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758158.934408:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758158.934415:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11215 00000100:00080000:0.0:1787758158.934418:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758158.934418:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758158.934425:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787758158.934428:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000008:00100000:1.0:1787758158.934431:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758158.934433:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655068544:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758158.934443:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655069312/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758158.934444:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655068672:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758158.934471:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758158.934480:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758158/real 1787758158] req@ffff8c05b2d9aa00 x1874588655068672/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758158.934488:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655068672/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758158.934493:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758158.934496:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758158.934498:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655068672:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758158.934501:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655068800:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758158.934512:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758158.934518:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758158/real 1787758158] req@ffff8c05b2d9bb80 x1874588655068800/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758158.934530:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655068800/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758158.934535:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758158.934537:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758158.934539:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655068800:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758158.934542:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655068928:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758158.934552:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758158.934558:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758158/real 1787758158] req@ffff8c05b2d98700 x1874588655068928/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758158.934565:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655068928/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758158.934570:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758158.934572:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758158.934574:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655068928:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758158.934585:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655069056:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758158.934595:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758158.934601:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758158/real 1787758158] req@ffff8c05b2d9b100 x1874588655069056/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758158.934608:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655069056/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758158.934617:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758158.934619:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758158.934621:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655069056:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758158.934624:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655069184:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758158.934633:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758158.934639:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758158/real 1787758158] req@ffff8c05b2d99880 x1874588655069184/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758158.934646:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655069184/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758158.934651:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758158.934653:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758158.934654:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655069184:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758158.934658:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655069312:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758158.934666:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758158.934671:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758158/real 1787758158] req@ffff8c05b2d99f80 x1874588655069312/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758158.934678:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655069312/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758158.934683:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758158.934685:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758158.934687:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655069312:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758159.061963:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758159.510073:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758160.085998:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758160.534003:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758161.109967:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758161.558092:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758162.133966:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758162.582008:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758163.157969:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758163.606130:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758164.053993:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758164.054004:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758164.054009:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758164.054020:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11220 00000100:00080000:0.0:1787758164.054031:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758164.054039:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758164.054083:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588655069440/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758164.054117:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758164.054121:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758164.054124:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758164.054129:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11220 00000100:00080000:0.0:1787758164.054133:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758164.054137:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758164.054148:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758164.054158:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655069440:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787758164.054162:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588655069568/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758164.054172:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758164.054175:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758164.054177:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758164.054181:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11220 00000100:00080000:0.0:1787758164.054184:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758164.054188:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758164.054193:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758164.054201:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655069696/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758164.054210:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758164.054214:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758164.054215:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758164.054218:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11220 00000100:00080000:0.0:1787758164.054222:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758164.054227:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758164.054232:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787758164.054235:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758164.054240:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588655069824/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758164.054248:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758164.054251:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758164.054253:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758164.054256:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11220 00000100:00080000:0.0:1787758164.054261:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758164.054265:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758164.054269:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758164.054278:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588655069952/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758164.054286:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758164.054287:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758164/real 1787758164] req@ffff8c05a9f74e00 x1874588655069440/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758164.054290:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758164.054292:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758164.054296:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11220 00000100:00080000:0.0:1787758164.054299:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758164.054303:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758164.054306:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588655069440/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787758164.054307:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758164.054317:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588655070080/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758164.054326:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758164.054327:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758164.054330:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758164.054331:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787758164.054332:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758164.054335:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11220 00000100:00100000:1.0:1787758164.054336:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655069440:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758164.054339:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758164.054352:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758164.054356:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758164.054374:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588655070208/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758164.054377:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655069568:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758164.054385:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758164.054389:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758164.054391:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758164.054395:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11220 00000100:00080000:0.0:1787758164.054399:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787758164.054400:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758164.054403:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758164.054408:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758164.054413:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758164/real 1787758164] req@ffff8c05a9f75500 x1874588655069568/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758164.054419:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588655070336/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758164.054426:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588655069568/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758164.054445:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758164.054448:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758164.054452:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655069568:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758164.054459:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655069696:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758164.054480:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758164.054491:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758164/real 1787758164] req@ffff8c05a9f75880 x1874588655069696/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758164.054499:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655069696/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758164.054505:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758164.054507:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758164.054509:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655069696:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758164.054514:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655069824:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758164.054528:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758164.054534:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758164/real 1787758164] req@ffff8c05a9f74000 x1874588655069824/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758164.054542:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588655069824/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758164.054548:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758164.054549:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758164.054551:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655069824:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758164.054554:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655069952:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758164.054565:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758164.054571:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758164/real 1787758164] req@ffff8c05a9f75180 x1874588655069952/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758164.054578:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588655069952/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758164.054584:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758164.054585:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758164.054587:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655069952:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758164.054591:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655070080:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758164.054602:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758164.054607:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758164/real 1787758164] req@ffff8c05a9f76d80 x1874588655070080/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758164.054615:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588655070080/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758164.054621:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758164.054622:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758164.054624:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655070080:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758164.054636:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655070208:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758164.054646:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758164.054652:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758164/real 1787758164] req@ffff8c05a9f77100 x1874588655070208/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758164.054659:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588655070208/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758164.054664:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758164.054666:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758164.054668:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655070208:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758164.054672:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655070336:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758164.054681:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758164.054687:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758164/real 1787758164] req@ffff8c05a9f74e00 x1874588655070336/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758164.054694:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588655070336/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758164.054713:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758164.054715:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758164.054717:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655070336:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758164.182000:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758164.630000:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758165.205962:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758165.654131:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758166.229956:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758166.678073:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758167.253945:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758167.702147:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758168.277939:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758168.726072:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758169.173953:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758169.173961:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758169.173964:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758169.173973:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11225 00000100:00080000:0.0:1787758169.173982:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758169.173988:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758169.174024:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655070464/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758169.174060:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758169.174064:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758169.174066:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758169.174070:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11225 00000100:00080000:0.0:1787758169.174074:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758169.174079:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758169.174088:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758169.174105:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655070464:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787758169.174107:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655070592/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758169.174122:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758169.174126:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758169.174128:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758169.174134:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11225 00000100:00080000:0.0:1787758169.174138:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758169.174143:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758169.174148:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758169.174159:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655070720/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758169.174170:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787758169.174171:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758169.174193:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758169.174197:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758169.174205:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11225 00000100:00080000:0.0:1787758169.174214:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758169.174221:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758169.174232:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758169.174263:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758169/real 1787758169] req@ffff8c05a9f74a80 x1874588655070464/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758169.174276:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588655070848/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758169.174295:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787758169.174303:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655070464/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758169.174304:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758169.174313:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758169.174331:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11225 00000100:00080000:2.0:1787758169.174332:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758169.174345:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758169.174345:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758169.174380:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655070464:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758169.174383:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758169.174395:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758169.174412:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655070592:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758169.174422:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588655070976/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758169.174453:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758169.174457:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:2.0:1787758169.174461:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758169.174462:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758169.174470:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11225 00000100:00080000:0.0:1787758169.174475:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787758169.174476:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758169/real 1787758169] req@ffff8c05a9f74380 x1874588655070592/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758169.174484:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758169.174491:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655070592/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787758169.174493:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787758169.174505:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758169.174508:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758169.174513:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588655071104/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758169.174513:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655070592:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758169.174523:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655070720:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758169.174535:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758169.174539:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758169.174542:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787758169.174544:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758169.174550:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11225 00000100:00080000:0.0:1787758169.174556:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787758169.174559:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758169/real 1787758169] req@ffff8c05a9f76300 x1874588655070720/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758169.174565:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758169.174575:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655070720/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787758169.174576:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787758169.174587:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758169.174591:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758169.174596:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655070720:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758169.174598:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655071232/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758169.174603:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655070848:192.168.201.107@tcp:8:kworker.0 00000100:00080000:0.0:1787758169.174618:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758169.174622:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758169.174625:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787758169.174626:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758169.174635:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11225 00000100:00080000:0.0:1787758169.174641:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787758169.174642:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758169/real 1787758169] req@ffff8c05a9f77b80 x1874588655070848/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758169.174650:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758169.174657:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588655070848/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787758169.174659:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787758169.174670:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787758169.174681:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655071360/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758169.174684:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758169.174691:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655070848:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758169.174721:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655070976:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758169.174737:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758169.174746:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758169/real 1787758169] req@ffff8c05a9f76a00 x1874588655070976/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758169.174758:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588655070976/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758169.174764:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758169.174766:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758169.174768:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655070976:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758169.174772:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655071104:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758169.174782:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758169.174788:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758169/real 1787758169] req@ffff8c05a9f75c00 x1874588655071104/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758169.174795:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588655071104/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758169.174800:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758169.174801:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758169.174803:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655071104:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758169.174806:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655071232:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758169.174816:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758169.174821:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758169/real 1787758169] req@ffff8c05a9f77800 x1874588655071232/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758169.174828:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655071232/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758169.174837:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758169.174840:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758169.174842:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655071232:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758169.174849:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655071360:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758169.174859:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758169.174864:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758169/real 1787758169] req@ffff8c05a9f76680 x1874588655071360/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758169.174872:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655071360/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758169.174913:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758169.174918:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758169.174927:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655071360:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758169.301963:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758169.750155:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758170.325963:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758170.774019:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758171.350017:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758171.798135:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758172.373972:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758172.822020:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758173.397978:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758173.846099:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758174.293989:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758174.293998:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758174.294002:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758174.294010:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11230 00000100:00080000:1.0:1787758174.294019:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758174.294025:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758174.294057:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655071488/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758174.294079:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758174.294082:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758174.294083:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758174.294086:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11230 00000100:00080000:1.0:1787758174.294089:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758174.294093:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758174.294102:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758174.294118:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655071616/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758174.294122:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655071488:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758174.294140:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758174.294144:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758174.294146:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758174.294151:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11230 00000100:00080000:1.0:1787758174.294155:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758174.294158:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758174.294163:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758174.294174:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655071744/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758174.294182:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758174.294184:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758174.294186:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787758174.294190:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758174.294194:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11230 00000100:00080000:1.0:1787758174.294198:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758174.294201:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758174.294205:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758174.294213:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655071872/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758174.294220:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758174.294222:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758174.294223:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758174.294226:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11230 00000100:00080000:1.0:1787758174.294229:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758174.294232:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758174.294237:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758174.294238:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758174/real 1787758174] req@ffff8c05b2d98000 x1874588655071488/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758174.294247:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655072000/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758174.294260:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758174.294263:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758174.294264:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758174.294269:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11230 00000100:00080000:1.0:1787758174.294271:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787758174.294271:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655071488/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758174.294277:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758174.294282:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787758174.294294:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787758174.294295:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655072128/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758174.294301:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758174.294309:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655071488:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758174.294313:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758174.294316:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758174.294318:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758174.294321:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11230 00000100:00080000:1.0:1787758174.294324:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758174.294326:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758174.294329:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758174.294339:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655072256/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758174.294339:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655071616:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758174.294360:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758174.294363:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758174.294364:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758174.294367:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11230 00000100:00080000:1.0:1787758174.294371:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758174.294374:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758174.294377:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787758174.294381:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758174.294389:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655072384/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758174.294397:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758174/real 1787758174] req@ffff8c05b2d9b480 x1874588655071616/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758174.294408:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655071616/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758174.294415:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758174.294418:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758174.294420:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655071616:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758174.294424:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655071744:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758174.294448:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758174.294456:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758174/real 1787758174] req@ffff8c05b2d99f80 x1874588655071744/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758174.294463:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655071744/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758174.294468:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758174.294470:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758174.294472:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655071744:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758174.294477:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655071872:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758174.294488:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758174.294494:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758174/real 1787758174] req@ffff8c05b2d99880 x1874588655071872/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758174.294501:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655071872/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758174.294506:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758174.294508:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758174.294510:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655071872:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758174.294513:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655072000:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758174.294531:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758174.294542:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758174/real 1787758174] req@ffff8c05b2d9b100 x1874588655072000/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758174.294555:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655072000/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758174.294564:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758174.294567:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758174.294571:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655072000:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758174.294586:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655072128:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758174.294605:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758174.294628:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758174/real 1787758174] req@ffff8c05b2d98700 x1874588655072128/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758174.294642:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655072128/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758174.294652:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758174.294656:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758174.294659:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655072128:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758174.294677:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655072256:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758174.294699:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758174.294712:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758174/real 1787758174] req@ffff8c05b2d9bb80 x1874588655072256/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758174.294724:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655072256/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758174.294731:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758174.294734:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758174.294737:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655072256:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758174.294743:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655072384:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758174.294762:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758174.294774:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758174/real 1787758174] req@ffff8c05b2d9aa00 x1874588655072384/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758174.294786:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655072384/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758174.294801:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758174.294803:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758174.294805:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655072384:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758174.421951:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758174.870012:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758175.445970:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758175.894115:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758176.470000:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758176.918030:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758177.493942:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758177.942061:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758178.517940:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758178.966030:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758179.413935:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758179.413943:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758179.413947:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758179.413957:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11235 00000100:00080000:1.0:1787758179.413969:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758179.413977:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758179.414015:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588655072512/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758179.414039:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758179.414043:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758179.414045:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758179.414049:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11235 00000100:00080000:1.0:1787758179.414054:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758179.414058:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758179.414068:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758179.414077:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655072512:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758179.414082:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655072640/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758179.414091:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758179.414095:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758179.414097:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758179.414100:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11235 00000100:00080000:1.0:1787758179.414103:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758179.414106:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758179.414111:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758179.414119:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655072768/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758179.414128:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758179.414131:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758179.414133:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758179.414136:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11235 00000100:00080000:1.0:1787758179.414139:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787758179.414141:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758179.414142:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758179.414147:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758179.414155:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655072896/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758179.414169:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758179.414173:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758179.414174:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758179.414178:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11235 00000100:00080000:1.0:1787758179.414181:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787758179.414183:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758179/real 1787758179] req@ffff8c05b2d9a300 x1874588655072512/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758179.414184:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758179.414189:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758179.414198:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655073024/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758179.414200:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588655072512/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758179.414208:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758179.414211:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758179.414211:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758179.414213:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758179.414215:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787758179.414217:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11235 00000100:00100000:2.0:1787758179.414219:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655072512:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758179.414220:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758179.414224:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758179.414228:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758179.414242:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655072640:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758179.414247:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655073152/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758179.414256:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758179.414260:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758179.414261:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787758179.414264:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758179.414265:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11235 00000100:00080000:1.0:1787758179.414269:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758179.414273:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758179.414276:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758179/real 1787758179] req@ffff8c05b2d98380 x1874588655072640/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787758179.414278:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758179.414288:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655073280/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758179.414289:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655072640/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758179.414297:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758179.414299:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758179.414301:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758179.414302:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787758179.414303:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758179.414306:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11235 00000100:00100000:2.0:1787758179.414306:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655072640:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758179.414310:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787758179.414312:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655072768:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758179.414313:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758179.414317:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758179.414326:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655073408/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787758179.414329:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758179.414339:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758179/real 1787758179] req@ffff8c05b2d9aa00 x1874588655072768/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758179.414351:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655072768/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758179.414359:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758179.414363:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758179.414367:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655072768:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758179.414373:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655072896:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758179.414392:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758179.414400:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758179/real 1787758179] req@ffff8c05b2d9bb80 x1874588655072896/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758179.414408:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655072896/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758179.414413:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758179.414415:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758179.414418:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655072896:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758179.414421:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655073024:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758179.414432:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758179.414438:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758179/real 1787758179] req@ffff8c05b2d98700 x1874588655073024/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758179.414445:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655073024/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758179.414450:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758179.414452:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758179.414454:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655073024:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758179.414465:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655073152:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758179.414475:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758179.414481:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758179/real 1787758179] req@ffff8c05b2d9b100 x1874588655073152/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758179.414489:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655073152/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758179.414493:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758179.414495:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758179.414497:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655073152:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758179.414500:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655073280:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758179.414509:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758179.414515:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758179/real 1787758179] req@ffff8c05b2d99880 x1874588655073280/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758179.414538:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655073280/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758179.414543:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758179.414545:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758179.414547:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655073280:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758179.414551:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655073408:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758179.414564:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758179.414570:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758179/real 1787758179] req@ffff8c05b2d99f80 x1874588655073408/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758179.414577:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655073408/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758179.414582:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758179.414584:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758179.414586:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655073408:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758179.541951:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758179.990067:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758180.565967:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758181.014042:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758181.589966:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758182.038104:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758182.613962:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758183.062025:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758183.637953:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758184.086093:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758184.533963:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758184.533971:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758184.533975:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758184.533983:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11240 00000100:00080000:1.0:1787758184.533992:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758184.533998:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758184.534030:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655073536/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758184.534058:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758184.534060:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758184.534062:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758184.534065:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11240 00000100:00080000:1.0:1787758184.534067:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758184.534069:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758184.534077:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758184.534088:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655073664/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758184.534095:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758184.534098:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758184.534099:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758184.534101:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11240 00000100:00080000:1.0:1787758184.534103:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787758184.534108:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655073536:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758184.534124:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758184.534129:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758184.534138:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655073792/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758184.534146:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758184.534149:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758184.534151:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758184.534154:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11240 00000100:00080000:1.0:1787758184.534164:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758184.534168:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:2.0:1787758184.534173:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:1.0:1787758184.534174:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758184.534184:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655073920/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758184.534191:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758184.534194:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758184.534195:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758184.534199:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11240 00000100:00080000:1.0:1787758184.534201:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758184.534205:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758184.534209:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758184.534215:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758184/real 1787758184] req@ffff8c05b2d9b480 x1874588655073536/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758184.534219:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655074048/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758184.534239:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758184.534242:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758184.534244:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787758184.534245:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655073536/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758184.534249:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11240 00000100:00080000:1.0:1787758184.534254:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758184.534259:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758184.534263:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787758184.534263:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758184.534272:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758184.534275:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655074176/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758184.534278:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655073536:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787758184.534290:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655073664:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758184.534302:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758184.534305:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758184.534307:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758184.534311:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11240 00000100:00080000:1.0:1787758184.534314:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787758184.534318:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758184.534321:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758184.534327:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758184.534332:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758184/real 1787758184] req@ffff8c05b2d98000 x1874588655073664/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758184.534337:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655074304/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758184.534346:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758184.534348:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758184.534349:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758184.534356:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11240 00000100:00080000:1.0:1787758184.534361:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787758184.534362:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655073664/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758184.534370:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:2.0:1787758184.534375:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787758184.534379:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787758184.534380:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758184.534386:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655073664:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758184.534399:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655074432/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758184.534409:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655073792:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758184.534427:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758184.534452:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758184/real 1787758184] req@ffff8c05b2d99f80 x1874588655073792/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758184.534461:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655073792/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758184.534471:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758184.534473:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758184.534475:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655073792:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758184.534479:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655073920:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758184.534492:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758184.534498:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758184/real 1787758184] req@ffff8c05b2d99880 x1874588655073920/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758184.534506:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655073920/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758184.534511:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758184.534513:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758184.534515:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655073920:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758184.534518:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655074048:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758184.534528:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758184.534533:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758184/real 1787758184] req@ffff8c05b2d9b100 x1874588655074048/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758184.534540:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655074048/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758184.534546:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758184.534547:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758184.534550:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655074048:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758184.534553:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655074176:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758184.534563:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758184.534568:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758184/real 1787758184] req@ffff8c05b2d98700 x1874588655074176/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758184.534575:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655074176/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758184.534580:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758184.534582:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758184.534583:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655074176:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758184.534586:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655074304:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758184.534595:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758184.534600:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758184/real 1787758184] req@ffff8c05b2d9bb80 x1874588655074304/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758184.534607:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655074304/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758184.534612:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758184.534614:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758184.534616:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655074304:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758184.534626:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655074432:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758184.534636:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758184.534642:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758184/real 1787758184] req@ffff8c05b2d9aa00 x1874588655074432/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758184.534650:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655074432/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758184.534656:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758184.534657:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758184.534659:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655074432:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758184.661979:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758185.110075:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758185.685924:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758186.134117:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758186.709948:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758187.158061:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758187.733951:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758188.182107:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758188.757953:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758189.206047:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758189.654003:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758189.654010:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787758189.654014:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758189.654022:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11246 00000100:00080000:2.0:1787758189.654031:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758189.654037:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758189.654068:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588655074560/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758189.654094:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758189.654097:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758189.654099:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758189.654102:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11246 00000100:00080000:2.0:1787758189.654111:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758189.654114:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758189.654121:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758189.654128:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655074560:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787758189.654134:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588655074688/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758189.654144:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758189.654146:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758189.654148:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758189.654150:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11246 00000100:00080000:2.0:1787758189.654153:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758189.654155:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758189.654159:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758189.654166:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588655074816/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758189.654173:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758189.654175:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758189.654176:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758189.654179:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11246 00000100:00080000:2.0:1787758189.654181:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758189.654183:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758189.654186:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787758189.654191:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758189.654195:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588655074944/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758189.654203:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758189.654205:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758189.654207:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758189.654210:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11246 00000100:00080000:2.0:1787758189.654214:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758189.654216:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758189.654219:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758189.654227:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588655075072/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758189.654232:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758189.654234:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758189.654235:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758189.654241:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11246 00000100:00100000:1.0:1787758189.654242:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758189/real 1787758189] req@ffff8c05b2e70e00 x1874588655074560/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758189.654243:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758189.654249:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758189.654256:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758189.654266:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588655075200/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758189.654269:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588655074560/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758189.654288:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758189.654291:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758189.654292:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758189.654293:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758189.654299:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787758189.654300:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11246 00000100:00100000:1.0:1787758189.654305:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655074560:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787758189.654308:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758189.654315:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758189.654320:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758189.654327:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655074688:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758189.654334:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588655075328/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758189.654367:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787758189.654370:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787758189.654373:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758189.654377:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758189.654383:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11246 00000100:00080000:2.0:1787758189.654386:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758189.654392:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758189.654393:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758189/real 1787758189] req@ffff8c05b2e70700 x1874588655074688/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787758189.654396:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758189.654410:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588655075456/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758189.654415:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588655074688/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758189.654428:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758189.654433:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758189.654437:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655074688:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758189.654445:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655074816:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758189.654468:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758189.654481:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758189/real 1787758189] req@ffff8c05b2e73480 x1874588655074816/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758189.654496:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588655074816/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758189.654502:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758189.654504:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758189.654506:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655074816:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758189.654511:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655074944:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758189.654527:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758189.654534:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758189/real 1787758189] req@ffff8c05b2e71500 x1874588655074944/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758189.654541:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588655074944/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758189.654546:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758189.654548:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758189.654550:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655074944:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758189.654553:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655075072:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758189.654563:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758189.654569:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758189/real 1787758189] req@ffff8c05b2e73b80 x1874588655075072/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758189.654576:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588655075072/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758189.654586:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758189.654588:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758189.654590:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655075072:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758189.654593:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655075200:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758189.654602:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758189.654608:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758189/real 1787758189] req@ffff8c05b2e71c00 x1874588655075200/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758189.654614:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588655075200/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758189.654619:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758189.654621:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758189.654623:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655075200:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758189.654645:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655075328:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758189.654657:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758189.654663:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758189/real 1787758189] req@ffff8c05b2e71880 x1874588655075328/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758189.654671:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588655075328/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758189.654676:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758189.654678:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758189.654681:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655075328:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758189.654684:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655075456:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758189.654694:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758189.654700:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758189/real 1787758189] req@ffff8c05b2e70380 x1874588655075456/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758189.654706:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588655075456/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758189.654712:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758189.654714:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758189.654715:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655075456:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758189.781949:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758190.230089:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758190.806020:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758191.254150:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758191.830018:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758192.278083:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758192.854013:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758193.302097:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758193.878014:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758194.326114:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758194.774098:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758194.774120:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758194.774128:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758194.774145:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11251 00000100:00080000:1.0:1787758194.774164:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758194.774175:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758194.774243:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655075584/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758194.774320:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758194.774325:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758194.774328:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758194.774335:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11251 00000100:00080000:1.0:1787758194.774341:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758194.774348:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758194.774363:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758194.774384:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588655075712/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758194.774407:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787758194.774408:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655075584:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758194.774424:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758194.774436:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758194.774445:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11251 00000100:00080000:1.0:1787758194.774450:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758194.774456:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758194.774468:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758194.774488:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655075840/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758194.774502:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758194.774507:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758194.774509:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758194.774516:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11251 00000100:00080000:1.0:1787758194.774520:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758194.774527:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:2.0:1787758194.774538:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:1.0:1787758194.774540:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758194.774564:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655075968/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758194.774580:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758194.774585:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758194.774588:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758194.774597:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11251 00000100:00080000:1.0:1787758194.774605:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758194.774612:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758194.774624:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758194/real 1787758194] req@ffff8c05b2d98380 x1874588655075584/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787758194.774627:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758194.774660:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655076096/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758194.774678:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655075584/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758194.774683:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758194.774705:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758194.774717:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758194.774741:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11251 00000100:00080000:2.0:1787758194.774745:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758194.774747:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758194.774770:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:2.0:1787758194.774772:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000008:00100000:1.0:1787758194.774789:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758194.774792:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655075584:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758194.774823:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655076224/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758194.774838:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655075712:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758194.774863:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758194.774870:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1787758194.774961:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758194.774980:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758194.774991:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11251 00000100:00080000:1.0:1787758194.775000:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787758194.775004:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758194/real 1787758194] req@ffff8c05b2d9a300 x1874588655075712/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758194.775012:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758194.775035:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758194.775059:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588655075712/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758194.775077:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655076352/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758194.775079:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758194.775106:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758194.775127:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655075712:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758194.775134:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758194.775146:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758194.775149:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758194.775163:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11251 00000100:00100000:2.0:1787758194.775165:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655075840:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758194.775170:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758194.775185:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758194.775195:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758194.775217:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655076480/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787758194.775247:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758194.775310:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758194/real 1787758194] req@ffff8c05b2d9aa00 x1874588655075840/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758194.775328:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655075840/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758194.775340:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758194.775345:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758194.775349:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655075840:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758194.775360:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655075968:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758194.775389:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758194.775402:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758194/real 1787758194] req@ffff8c05b2d9bb80 x1874588655075968/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758194.775416:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655075968/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758194.775435:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758194.775439:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758194.775443:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655075968:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758194.775451:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655076096:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758194.775471:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758194.775482:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758194/real 1787758194] req@ffff8c05b2d98700 x1874588655076096/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758194.775496:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655076096/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758194.775507:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758194.775510:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758194.775514:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655076096:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758194.775540:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655076224:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758194.775561:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758194.775573:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758194/real 1787758194] req@ffff8c05b2d9b100 x1874588655076224/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758194.775587:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655076224/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758194.775597:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758194.775600:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758194.775605:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655076224:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758194.775613:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655076352:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758194.775632:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758194.775643:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758194/real 1787758194] req@ffff8c05b2d99880 x1874588655076352/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758194.775656:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655076352/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758194.775667:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758194.775670:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758194.775674:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655076352:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758194.775680:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655076480:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758194.775698:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758194.775709:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758194/real 1787758194] req@ffff8c05b2d99f80 x1874588655076480/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758194.775722:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655076480/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758194.775732:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758194.775736:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758194.775740:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655076480:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758194.902011:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758195.350130:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758195.926027:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758196.374242:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758196.950014:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758197.398142:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758197.974075:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758198.422253:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758198.997970:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758199.446153:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758199.893982:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758199.893990:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758199.893994:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758199.894001:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11256 00000100:00080000:3.0:1787758199.894010:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758199.894016:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758199.894050:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655076608/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758199.894073:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758199.894075:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758199.894077:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758199.894080:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11256 00000100:00080000:3.0:1787758199.894082:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758199.894084:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758199.894091:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758199.894099:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655076736/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758199.894111:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758199.894121:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758199.894123:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758199.894126:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11256 00000100:00080000:3.0:1787758199.894129:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758199.894132:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758199.894137:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758199.894145:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655076864/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758199.894154:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758199.894158:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758199.894174:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758199.894180:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11256 00000100:00100000:2.0:1787758199.894181:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655076608:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758199.894184:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758199.894195:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758199.894201:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758199.894209:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655076992/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758199.894217:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758199.894220:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758199.894221:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758199.894224:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11256 00000100:00080000:3.0:1787758199.894227:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758199.894230:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758199.894233:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758199.894241:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655077120/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758199.894247:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758199.894250:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758199.894251:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758199.894254:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11256 00000100:00080000:3.0:1787758199.894277:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758199.894279:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758199.894283:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758199.894291:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655077248/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758199.894297:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758199.894300:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758199.894302:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758199.894304:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11256 00000100:00080000:3.0:1787758199.894306:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758199.894308:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758199.894311:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758199.894318:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655077376/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758199.894329:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758199.894332:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758199.894335:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758199.894339:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11256 00000100:00080000:3.0:1787758199.894346:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758199.894349:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758199.894351:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758199.894366:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655076736:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758199.894366:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655077504/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758199.894428:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655076864:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758199.894457:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655076992:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758199.894475:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655077120:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758199.894503:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655077248:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758199.894520:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655077376:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758199.894535:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655077504:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758200.022015:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758200.470242:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758201.046038:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758201.494229:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758202.070034:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758202.518282:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787758202.966151:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.107:988 00000400:00000100:0.0:1787758202.966197:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.107@tcp at host 192.168.201.107 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787758202.966212:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787758202.966248:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787758202.966254:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787758202.966261:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787758202.966269:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787758202.966276:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787758202.966296:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787758202.966304:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:2.0:1787758202.966336:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758199/real 1787758202] req@ffff8c05a9f79180 x1874588655076864/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758202.966401:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655076864/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758202.966427:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758202.966436:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758202.966445:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655076864:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758202.966464:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758199/real 1787758202] req@ffff8c05a9f7b100 x1874588655076992/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758202.966491:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655076992/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758202.966508:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758202.966515:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758202.966523:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655076992:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758202.966538:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758199/real 1787758202] req@ffff8c05a9f7ad80 x1874588655077120/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758202.966563:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655077120/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758202.966580:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758202.966587:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758202.966596:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655077120:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758202.966611:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758199/real 1787758202] req@ffff8c05a9f78e00 x1874588655077248/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758202.966636:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655077248/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758202.966654:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758202.966662:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758202.966670:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655077248:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758202.966699:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758199/real 1787758202] req@ffff8c05a9f7b800 x1874588655077376/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758202.966727:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655077376/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758202.966753:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758202.966760:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758202.966769:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655077376:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758202.966784:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758199/real 1787758202] req@ffff8c05a9f79500 x1874588655077504/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758202.966807:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655077504/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758202.966821:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758202.966827:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758202.966834:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655077504:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758202.966848:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758199/real 1787758202] req@ffff8c05a9f7bb80 x1874588655076608/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758202.966870:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655076608/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758202.966953:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758202.966966:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758202.966975:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655076608:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787758202.966991:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758199/real 1787758202] req@ffff8c05a9f79f80 x1874588655076736/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758202.967016:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655076736/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758202.967032:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758202.967040:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758202.967048:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655076736:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758203.093981:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758203.542148:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758204.118015:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758204.566360:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758205.014153:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758205.014166:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787758205.014172:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758205.014197:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11261 00000100:00080000:2.0:1787758205.014212:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758205.014223:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758205.014270:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73800 x1874588655077632/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758205.014300:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758205.014305:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758205.014309:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758205.014318:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11261 00000100:00080000:2.0:1787758205.014325:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758205.014334:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758205.014349:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758205.014375:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588655077760/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758205.014390:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758205.014395:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758205.014398:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758205.014410:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11261 00000100:00080000:2.0:1787758205.014419:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758205.014427:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758205.014439:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655077632:192.168.201.107@tcp:250:kworker.0 00000008:00100000:2.0:1787758205.014441:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758205.014470:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588655077888/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758205.014490:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758205.014494:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758205.014497:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758205.014505:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11261 00000100:00080000:2.0:1787758205.014512:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758205.014521:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758205.014530:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758205.014552:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588655078016/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787758205.014572:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787758205.014577:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758205.014586:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758205.014590:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758205.014598:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11261 00000100:00080000:2.0:1787758205.014606:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758205.014613:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758205.014621:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758205.014653:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758205/real 1787758205] req@ffff8c05b2e73800 x1874588655077632/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758205.014654:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588655078144/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758205.014685:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758205.014692:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758205.014694:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758205.014708:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11261 00000100:00080000:2.0:1787758205.014715:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758205.014722:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758205.014730:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73800 x1874588655077632/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787758205.014751:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787758205.014752:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758205.014764:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758205.014772:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655077632:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758205.014793:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655077760:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758205.014795:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588655078272/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758205.014821:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758205.014828:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787758205.014831:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758205.014848:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11261 00000800:00000100:1.0:1787758205.014850:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787758205.014861:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758205.014872:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758205.014928:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758205/real 1787758205] req@ffff8c05b2e70380 x1874588655077760/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758205.014952:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588655077760/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787758205.014972:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787758205.014976:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758205.014986:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758205.014992:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655077760:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758205.015013:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588655078400/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758205.015030:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758205.015035:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787758205.015046:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655077888:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787758205.015046:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758205.015060:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11261 00000100:00080000:2.0:1787758205.015090:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758205.015098:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758205.015107:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758205.015137:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588655078528/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787758205.015150:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758205.015194:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758205/real 1787758205] req@ffff8c05b2e71880 x1874588655077888/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758205.015218:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588655077888/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758205.015227:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758205.015230:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758205.015233:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655077888:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758205.015242:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655078016:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758205.015266:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758205.015277:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758205/real 1787758205] req@ffff8c05b2e71c00 x1874588655078016/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758205.015292:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588655078016/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758205.015303:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758205.015308:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758205.015312:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655078016:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758205.015321:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655078144:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758205.015350:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758205.015364:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758205/real 1787758205] req@ffff8c05b2e73b80 x1874588655078144/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758205.015381:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588655078144/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758205.015393:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758205.015396:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758205.015399:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655078144:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758205.015408:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655078272:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758205.015437:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758205.015451:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758205/real 1787758205] req@ffff8c05b2e71500 x1874588655078272/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758205.015467:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588655078272/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758205.015479:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758205.015484:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758205.015488:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655078272:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758205.015496:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655078400:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758205.015521:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758205.015536:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758205/real 1787758205] req@ffff8c05b2e73480 x1874588655078400/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758205.015550:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588655078400/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758205.015560:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758205.015565:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758205.015568:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655078400:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758205.015603:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655078528:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758205.015638:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758205.015653:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758205/real 1787758205] req@ffff8c05b2e70700 x1874588655078528/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758205.015663:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588655078528/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758205.015677:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758205.015680:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758205.015683:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655078528:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758205.142021:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758205.590074:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758206.166030:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758206.614361:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758207.190019:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758207.638179:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758208.213961:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758208.662140:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758209.237986:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758209.686024:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758210.134023:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758210.134034:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758210.134037:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758210.134046:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11266 00000100:00080000:3.0:1787758210.134055:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758210.134061:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758210.134094:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655078656/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758210.134118:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758210.134121:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758210.134122:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758210.134126:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11266 00000100:00080000:3.0:1787758210.134128:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758210.134130:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758210.134138:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758210.134148:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655078784/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758210.134156:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758210.134160:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787758210.134161:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655078656:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758210.134162:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758210.134169:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11266 00000100:00080000:3.0:1787758210.134172:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758210.134174:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758210.134178:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758210.134186:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655078912/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758210.134197:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758210.134200:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758210.134201:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758210.134204:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11266 00000100:00080000:3.0:1787758210.134207:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758210.134209:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758210.134213:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758210.134221:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655079040/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787758210.134228:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758210.134234:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758210.134237:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758210.134239:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758210.134242:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11266 00000100:00080000:3.0:1787758210.134246:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758210.134249:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758210.134270:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758210.134289:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655079168/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758210.134303:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758210/real 1787758210] req@ffff8c05a9f7b480 x1874588655078656/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758210.134305:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758210.134321:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758210.134326:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758210.134343:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11266 00000100:00080000:3.0:1787758210.134351:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758210.134357:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758210.134364:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655078656/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787758210.134367:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787758210.134392:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787758210.134393:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655079296/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758210.134407:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787758210.134423:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758210.134426:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655078656:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758210.134429:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758210.134435:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758210.134451:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11266 00000100:00080000:3.0:1787758210.134455:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758210.134459:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758210.134465:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758210.134476:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655079424/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758210.134492:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655078784:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758210.134493:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758210.134502:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758210.134508:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758210.134519:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11266 00000100:00080000:3.0:1787758210.134522:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758210.134527:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758210.134533:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758210.134553:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655079552/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787758210.134570:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758210.134605:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758210/real 1787758210] req@ffff8c05a9f78380 x1874588655078784/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758210.134628:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655078784/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758210.134657:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758210.134663:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758210.134668:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655078784:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758210.134677:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655078912:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758210.134718:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758210.134726:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758210/real 1787758210] req@ffff8c05a9f78a80 x1874588655078912/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758210.134734:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655078912/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758210.134739:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758210.134742:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758210.134744:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655078912:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758210.134748:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655079040:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758210.134763:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758210.134798:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758210/real 1787758210] req@ffff8c05a9f7a680 x1874588655079040/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758210.134822:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655079040/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758210.134851:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758210.134855:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758210.134859:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655079040:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758210.134865:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655079168:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758210.134880:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758210.134888:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758210/real 1787758210] req@ffff8c05a9f7aa00 x1874588655079168/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758210.134896:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655079168/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758210.134901:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758210.134903:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758210.134906:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655079168:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758210.134912:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655079296:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758210.134957:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758210.134968:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758210/real 1787758210] req@ffff8c05a9f7a300 x1874588655079296/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758210.134976:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655079296/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758210.134999:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758210.135001:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758210.135004:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655079296:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758210.135018:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655079424:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758210.135032:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758210.135040:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758210/real 1787758210] req@ffff8c05a9f79880 x1874588655079424/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758210.135058:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655079424/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758210.135067:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758210.135070:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758210.135074:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655079424:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758210.135080:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655079552:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758210.135101:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758210.135110:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758210/real 1787758210] req@ffff8c05a9f78700 x1874588655079552/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758210.135118:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655079552/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758210.135124:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758210.135126:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758210.135128:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655079552:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758210.261956:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758210.710262:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758211.286005:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758211.734010:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758212.309987:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758212.758138:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758213.334015:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758213.782030:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758214.357985:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758214.806107:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758215.254002:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758215.254010:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758215.254014:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758215.254021:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11271 00000100:00080000:1.0:1787758215.254030:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758215.254037:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758215.254069:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655079680/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758215.254095:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758215.254098:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758215.254099:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758215.254102:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11271 00000100:00080000:1.0:1787758215.254104:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758215.254106:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758215.254114:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758215.254123:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655079808/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758215.254132:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758215.254135:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758215.254137:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758215.254142:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11271 00000100:00100000:2.0:1787758215.254147:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655079680:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758215.254148:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758215.254158:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758215.254164:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758215.254180:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655079936/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758215.254192:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758215.254196:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758215.254199:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758215.254203:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11271 00000100:00080000:1.0:1787758215.254206:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758215.254212:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:2.0:1787758215.254213:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:1.0:1787758215.254221:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758215.254231:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655080064/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758215.254238:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758215.254240:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758215.254242:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758215.254245:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11271 00000100:00080000:1.0:1787758215.254247:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758215.254250:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758215.254253:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758215.254256:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758215/real 1787758215] req@ffff8c05b2d98000 x1874588655079680/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758215.254263:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655080192/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758215.254272:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758215.254277:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787758215.254277:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655079680/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758215.254279:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758215.254287:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11271 00000100:00080000:1.0:1787758215.254292:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758215.254296:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:2.0:1787758215.254296:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787758215.254299:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787758215.254303:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758215.254308:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655079680:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758215.254311:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655080320/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758215.254318:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655079808:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758215.254322:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758215.254325:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758215.254327:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758215.254330:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11271 00000100:00080000:1.0:1787758215.254333:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758215.254336:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758215.254340:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787758215.254343:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758215.254352:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655080448/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758215.254357:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758215/real 1787758215] req@ffff8c05b2d9b480 x1874588655079808/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758215.254363:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758215.254370:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758215.254373:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758215.254381:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11271 00000100:00080000:1.0:1787758215.254384:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787758215.254384:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655079808/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758215.254388:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758215.254395:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787758215.254399:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787758215.254405:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655080576/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758215.254406:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758215.254413:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655079808:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758215.254442:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655079936:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758215.254456:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758215.254463:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758215/real 1787758215] req@ffff8c05b2d99f80 x1874588655079936/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758215.254471:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655079936/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758215.254476:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758215.254478:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758215.254480:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655079936:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758215.254484:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655080064:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758215.254493:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758215.254499:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758215/real 1787758215] req@ffff8c05b2d99880 x1874588655080064/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758215.254506:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655080064/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758215.254512:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758215.254513:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758215.254515:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655080064:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758215.254518:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655080192:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758215.254528:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758215.254534:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758215/real 1787758215] req@ffff8c05b2d9b100 x1874588655080192/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758215.254546:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655080192/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758215.254551:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758215.254552:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758215.254554:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655080192:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758215.254557:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655080320:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758215.254566:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758215.254572:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758215/real 1787758215] req@ffff8c05b2d98700 x1874588655080320/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758215.254578:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655080320/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758215.254583:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758215.254585:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758215.254587:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655080320:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758215.254590:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655080448:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758215.254600:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758215.254605:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758215/real 1787758215] req@ffff8c05b2d9bb80 x1874588655080448/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758215.254612:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655080448/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758215.254631:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758215.254634:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758215.254636:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655080448:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758215.254639:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655080576:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758215.254648:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758215.254654:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758215/real 1787758215] req@ffff8c05b2d9aa00 x1874588655080576/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758215.254661:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655080576/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758215.254666:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758215.254668:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758215.254670:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655080576:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758215.381972:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758215.830004:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758216.406043:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758216.854179:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758217.429979:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758217.877996:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758218.454091:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758218.902086:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758219.477961:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758219.926055:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758220.374008:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758220.374020:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758220.374025:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758220.374036:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11276 00000100:00080000:3.0:1787758220.374048:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758220.374055:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758220.374105:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655080704/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758220.374133:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758220.374137:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758220.374138:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758220.374142:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11276 00000100:00080000:3.0:1787758220.374145:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758220.374149:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758220.374158:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758220.374168:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655080832/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758220.374176:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758220.374181:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787758220.374183:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655080704:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758220.374184:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758220.374195:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11276 00000100:00080000:3.0:1787758220.374200:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758220.374203:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758220.374207:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758220.374221:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655080960/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758220.374228:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758220.374238:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758220.374239:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758220.374246:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11276 00000100:00080000:3.0:1787758220.374257:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758220.374261:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758220.374265:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787758220.374270:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758220.374275:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655081088/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758220.374283:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758220.374288:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758220.374289:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758220.374294:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11276 00000100:00080000:3.0:1787758220.374297:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758220.374301:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758220.374306:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758220.374316:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655081216/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758220.374325:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758220.374330:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758220.374332:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787758220.374334:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758220/real 1787758220] req@ffff8c05a9f78700 x1874588655080704/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758220.374336:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11276 00000100:00080000:3.0:1787758220.374343:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758220.374354:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758220.374366:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758220.374383:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655080704/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758220.374383:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655081344/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758220.374459:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758220.374461:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758220.374468:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758220.374471:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758220.374473:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758220.374493:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655080704:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758220.374493:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11276 00000100:00080000:3.0:1787758220.374504:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758220.374509:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758220.374514:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758220.374527:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655081472/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758220.374530:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655080832:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758220.374555:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758220.374559:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758220.374561:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758220.374566:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11276 00000100:00080000:3.0:1787758220.374569:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787758220.374577:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758220.374578:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758220.374586:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758220.374599:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655081600/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758220.374609:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758220/real 1787758220] req@ffff8c05a9f79880 x1874588655080832/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758220.374626:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655080832/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758220.374634:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758220.374638:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758220.374641:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655080832:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758220.374647:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655080960:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758220.374665:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758220.374672:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758220/real 1787758220] req@ffff8c05a9f7a300 x1874588655080960/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758220.374680:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655080960/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758220.374685:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758220.374687:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758220.374689:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655080960:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758220.374693:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655081088:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758220.374704:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758220.374710:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758220/real 1787758220] req@ffff8c05a9f7aa00 x1874588655081088/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758220.374718:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655081088/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758220.374723:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758220.374725:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758220.374727:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655081088:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758220.374731:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655081216:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758220.374741:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758220.374746:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758220/real 1787758220] req@ffff8c05a9f7a680 x1874588655081216/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758220.374754:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655081216/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758220.374759:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758220.374760:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758220.374762:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655081216:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758220.374766:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655081344:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758220.374775:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758220.374781:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758220/real 1787758220] req@ffff8c05a9f78a80 x1874588655081344/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758220.374788:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655081344/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758220.374798:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758220.374814:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758220.374817:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655081344:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758220.374832:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655081472:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758220.374847:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758220.374855:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758220/real 1787758220] req@ffff8c05a9f78380 x1874588655081472/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758220.374862:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655081472/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758220.374868:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758220.374870:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758220.374872:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655081472:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758220.374899:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655081600:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758220.374913:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758220.374920:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758220/real 1787758220] req@ffff8c05a9f7b480 x1874588655081600/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758220.374928:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655081600/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758220.374934:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758220.374936:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758220.374938:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655081600:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758220.501975:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758220.950084:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758221.525953:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758221.974080:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758222.549951:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758222.998070:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758223.573948:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758224.021979:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758224.597935:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758225.046105:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758225.493975:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758225.493986:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758225.493991:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758225.494002:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11281 00000100:00080000:1.0:1787758225.494013:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758225.494021:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758225.494058:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588655081728/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758225.494093:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758225.494101:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758225.494105:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758225.494113:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11281 00000100:00080000:1.0:1787758225.494119:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758225.494127:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758225.494137:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758225.494155:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655081856/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758225.494164:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655081728:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758225.494192:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758225.494197:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758225.494200:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758225.494205:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11281 00000100:00080000:1.0:1787758225.494211:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758225.494216:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758225.494222:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787758225.494236:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758225.494239:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655081984/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758225.494259:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758225.494262:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758225.494265:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758225.494269:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11281 00000100:00100000:2.0:1787758225.494274:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758225/real 1787758225] req@ffff8c05b2d9a300 x1874588655081728/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758225.494275:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758225.494284:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758225.494292:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758225.494298:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588655081728/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758225.494308:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787758225.494310:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655082112/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758225.494314:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758225.494322:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655081728:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758225.494331:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758225.494336:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758225.494339:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758225.494345:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11281 00000100:00080000:1.0:1787758225.494351:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787758225.494351:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655081856:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758225.494361:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758225.494368:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787758225.494381:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758225.494387:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655082240/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758225.494399:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758225/real 1787758225] req@ffff8c05b2d98380 x1874588655081856/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758225.494402:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758225.494411:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758225.494415:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787758225.494420:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655081856/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758225.494424:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11281 00000100:00080000:1.0:1787758225.494433:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758225.494436:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758225.494440:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:2.0:1787758225.494442:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758225.494449:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655081856:192.168.201.107@tcp:38:kworker.0 00000008:00100000:1.0:1787758225.494453:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758225.494456:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655081984:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758225.494473:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655082368/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787758225.494482:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758225.494492:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787758225.494497:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758225/real 1787758225] req@ffff8c05b2d9aa00 x1874588655081984/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758225.494498:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758225.494503:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758225.494513:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11281 00000100:00080000:1.0:1787758225.494518:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787758225.494521:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655081984/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758225.494528:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:2.0:1787758225.494531:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758225.494537:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000008:00100000:1.0:1787758225.494538:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758225.494542:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655081984:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758225.494550:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655082112:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758225.494558:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655082496/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787758225.494568:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758225.494577:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787758225.494580:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758225/real 1787758225] req@ffff8c05b2d9bb80 x1874588655082112/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758225.494583:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758225.494587:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758225.494596:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11281 00000100:00100000:2.0:1787758225.494598:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655082112/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758225.494604:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758225.494606:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758225.494612:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787758225.494615:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758225.494618:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655082112:192.168.201.107@tcp:8:kworker.0 00000008:00100000:1.0:1787758225.494625:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758225.494639:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655082624/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758225.494646:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655082240:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758225.494668:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758225.494677:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758225/real 1787758225] req@ffff8c05b2d98700 x1874588655082240/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758225.494685:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655082240/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758225.494691:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758225.494693:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758225.494695:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655082240:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758225.494698:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655082368:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758225.494708:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758225.494728:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758225/real 1787758225] req@ffff8c05b2d9b100 x1874588655082368/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758225.494736:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655082368/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758225.494741:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758225.494743:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758225.494745:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655082368:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758225.494749:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655082496:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758225.494760:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758225.494766:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758225/real 1787758225] req@ffff8c05b2d99880 x1874588655082496/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758225.494773:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655082496/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758225.494778:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758225.494780:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758225.494783:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655082496:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758225.494791:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655082624:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758225.494802:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758225.494808:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758225/real 1787758225] req@ffff8c05b2d99f80 x1874588655082624/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758225.494815:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655082624/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758225.494820:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758225.494821:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758225.494823:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655082624:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758225.621945:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758226.070020:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758226.645957:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758227.094102:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758227.669958:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758228.118042:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758228.693978:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758229.142214:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758229.717974:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758230.166050:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758230.614025:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758230.614036:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758230.614041:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758230.614057:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11287 00000100:00080000:3.0:1787758230.614073:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758230.614083:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758230.614136:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655082752/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758230.614166:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758230.614169:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758230.614170:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758230.614174:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11287 00000100:00080000:3.0:1787758230.614177:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758230.614180:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758230.614189:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758230.614198:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655082880/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758230.614207:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758230.614211:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758230.614212:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758230.614215:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11287 00000100:00080000:3.0:1787758230.614218:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758230.614221:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758230.614225:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758230.614231:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655083008/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758230.614243:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655082752:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758230.614269:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758230.614273:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758230.614277:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758230.614286:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11287 00000100:00080000:3.0:1787758230.614290:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758230.614294:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758230.614300:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758230.614313:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655083136/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758230.614323:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758230.614326:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758230.614328:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758230.614338:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11287 00000800:00000100:2.0:1787758230.614341:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758230.614349:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758230.614356:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758230.614361:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758230.614370:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655083264/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758230.614379:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758230.614382:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758230.614383:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758230.614386:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11287 00000100:00080000:3.0:1787758230.614389:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758230.614392:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758230.614396:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758230.614404:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758230/real 1787758230] req@ffff8c05a9f79f80 x1874588655082752/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758230.614405:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655083392/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758230.614484:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758230.614487:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758230.614489:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787758230.614496:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655082752/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758230.614501:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11287 00000100:00080000:3.0:1787758230.614510:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758230.614523:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:2.0:1787758230.614532:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787758230.614532:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787758230.614544:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758230.614555:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655082752:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787758230.614555:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655083520/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758230.614582:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655082880:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758230.614584:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758230.614597:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758230.614600:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758230.614603:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11287 00000100:00080000:3.0:1787758230.614607:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758230.614611:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758230.614615:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787758230.614631:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758230.614645:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655083648/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758230.614650:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758230/real 1787758230] req@ffff8c05a9f7bb80 x1874588655082880/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758230.614744:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655082880/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758230.614752:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758230.614755:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758230.614758:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655082880:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758230.614782:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655083008:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758230.614796:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758230.614803:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758230/real 1787758230] req@ffff8c05a9f79500 x1874588655083008/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758230.614811:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655083008/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758230.614816:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758230.614818:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758230.614820:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655083008:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758230.614824:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655083136:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758230.614835:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758230.614841:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758230/real 1787758230] req@ffff8c05a9f7b800 x1874588655083136/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758230.614848:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655083136/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758230.614854:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758230.614856:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758230.614858:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655083136:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758230.614862:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655083264:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758230.614871:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758230.614904:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758230/real 1787758230] req@ffff8c05a9f78e00 x1874588655083264/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758230.614913:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655083264/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758230.614918:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758230.614921:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758230.614922:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655083264:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758230.614931:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655083392:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758230.614944:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758230.614951:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758230/real 1787758230] req@ffff8c05a9f7ad80 x1874588655083392/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758230.614959:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655083392/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758230.614964:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758230.614966:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758230.614969:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655083392:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758230.614973:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655083520:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758230.614983:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758230.614989:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758230/real 1787758230] req@ffff8c05a9f7b100 x1874588655083520/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758230.614996:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655083520/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758230.615001:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758230.615003:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758230.615005:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655083520:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758230.615008:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655083648:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758230.615017:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758230.615023:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758230/real 1787758230] req@ffff8c05a9f79180 x1874588655083648/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758230.615030:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655083648/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758230.615035:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758230.615037:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758230.615039:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655083648:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758230.741999:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758231.190112:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758231.766008:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758232.214063:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758232.789957:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758233.238092:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758233.813966:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758234.262017:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758234.837974:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758235.286071:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758235.733984:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758235.733995:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787758235.734001:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758235.734011:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11292 00000100:00080000:2.0:1787758235.734022:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758235.734030:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758235.734073:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588655083776/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758235.734102:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758235.734107:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758235.734109:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758235.734114:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11292 00000100:00080000:2.0:1787758235.734118:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758235.734123:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758235.734135:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758235.734148:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588655083904/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758235.734149:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655083776:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787758235.734159:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758235.734163:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758235.734165:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758235.734168:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11292 00000100:00080000:2.0:1787758235.734172:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758235.734176:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758235.734181:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758235.734198:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588655084032/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758235.734213:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758235.734217:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758235.734218:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758235.734222:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11292 00000100:00080000:2.0:1787758235.734226:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758235.734230:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758235.734235:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787758235.734237:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758235.734246:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588655084160/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758235.734254:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758235.734258:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758235.734259:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758235.734263:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11292 00000100:00080000:2.0:1787758235.734267:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758235.734271:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758235.734275:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758235.734284:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588655084288/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758235.734287:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758235/real 1787758235] req@ffff8c05b2e70e00 x1874588655083776/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758235.734304:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758235.734307:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588655083776/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758235.734307:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758235.734309:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758235.734314:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11292 00000100:00080000:2.0:1787758235.734318:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758235.734319:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758235.734322:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:1.0:1787758235.734324:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000008:00100000:2.0:1787758235.734327:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758235.734328:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655083776:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787758235.734337:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588655084416/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758235.734352:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758235.734356:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655083904:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787758235.734356:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787758235.734358:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758235.734362:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11292 00000100:00080000:2.0:1787758235.734365:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758235.734369:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758235.734374:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787758235.734378:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758235.734385:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588655084544/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758235.734392:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758235/real 1787758235] req@ffff8c05b2e70000 x1874588655083904/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758235.734403:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758235.734405:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588655083904/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758235.734407:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758235.734409:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758235.734413:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11292 00000100:00080000:1.0:1787758235.734414:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758235.734416:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758235.734418:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787758235.734421:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758235.734423:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655083904:192.168.201.107@tcp:38:kworker.0 00000008:00100000:2.0:1787758235.734425:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758235.734429:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655084032:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758235.734435:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588655084672/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787758235.734449:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758235.734471:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758235/real 1787758235] req@ffff8c05b2e72300 x1874588655084032/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758235.734484:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588655084032/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758235.734500:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758235.734502:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758235.734505:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655084032:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758235.734510:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655084160:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758235.734527:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758235.734550:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758235/real 1787758235] req@ffff8c05b2e72a00 x1874588655084160/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758235.734558:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588655084160/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758235.734564:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758235.734566:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758235.734569:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655084160:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758235.734572:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655084288:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758235.734585:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758235.734593:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758235/real 1787758235] req@ffff8c05b2e73100 x1874588655084288/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758235.734600:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588655084288/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758235.734606:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758235.734607:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758235.734609:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655084288:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758235.734612:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655084416:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758235.734623:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758235.734628:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758235/real 1787758235] req@ffff8c05b2e72d80 x1874588655084416/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758235.734636:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588655084416/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758235.734641:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758235.734643:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758235.734645:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655084416:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758235.734657:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655084544:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758235.734667:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758235.734673:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758235/real 1787758235] req@ffff8c05b2e72680 x1874588655084544/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758235.734681:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588655084544/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758235.734686:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758235.734689:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758235.734691:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655084544:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758235.734694:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655084672:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758235.734703:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758235.734708:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758235/real 1787758235] req@ffff8c05b2e71180 x1874588655084672/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758235.734715:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588655084672/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758235.734721:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758235.734723:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758235.734725:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655084672:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758235.862011:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758236.310043:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758236.885973:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758237.334049:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758237.909953:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758238.358022:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758238.933969:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758239.382196:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758239.957957:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758240.406156:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758240.854066:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758240.854080:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758240.854086:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758240.854099:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11297 00000100:00080000:3.0:1787758240.854113:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758240.854122:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758240.854174:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655084800/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758240.854219:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758240.854226:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758240.854231:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758240.854239:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11297 00000100:00080000:3.0:1787758240.854247:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758240.854256:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758240.854272:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758240.854304:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655084928/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758240.854310:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655084800:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758240.854352:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758240.854358:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758240.854362:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758240.854374:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11297 00000100:00080000:3.0:1787758240.854382:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758240.854389:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758240.854398:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758240.854418:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655085056/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787758240.854436:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758240.854437:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758240.854471:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758240.854474:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758240.854494:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11297 00000100:00080000:3.0:1787758240.854502:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758240.854510:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758240.854519:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758240.854543:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655085184/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758240.854554:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758240/real 1787758240] req@ffff8c05a9f79180 x1874588655084800/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758240.854575:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655084800/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758240.854586:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758240.854590:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758240.854593:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655084800:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787758240.854620:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655084928:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758240.854650:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758240.854662:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758240/real 1787758240] req@ffff8c05a9f7b100 x1874588655084928/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758240.854673:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655084928/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758240.854680:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758240.854686:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758240.854697:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655084928:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758240.854713:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655085056:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758240.854733:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758240.854751:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758240/real 1787758240] req@ffff8c05a9f7ad80 x1874588655085056/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758240.854760:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655085056/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758240.854766:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758240.854769:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758240.854772:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655085056:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758240.855107:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758240.855116:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758240.855121:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758240.855131:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11297 00000100:00080000:3.0:1787758240.855141:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787758240.855145:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655085184:192.168.201.107@tcp:8:kworker.0 00000100:00080000:3.0:1787758240.855155:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758240.855167:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787758240.855181:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758240.855192:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655085312/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758240.855202:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758240/real 1787758240] req@ffff8c05a9f78e00 x1874588655085184/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758240.855221:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787758240.855229:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655085184/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758240.855236:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758240.855245:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758240.855246:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758240.855255:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758240.855265:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655085184:192.168.201.107@tcp:8:kworker.0 00000100:00080000:3.0:1787758240.855267:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11297 00000100:00080000:3.0:1787758240.855287:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787758240.855296:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655085312:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758240.855306:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758240.855321:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787758240.855324:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758240.855346:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758240/real 1787758240] req@ffff8c05a9f7b800 x1874588655085312/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758240.855356:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655085440/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758240.855365:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655085312/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758240.855395:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758240.855396:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758240.855418:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787758240.855424:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787758240.855428:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787758240.855432:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655085312:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758240.855467:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11297 00000100:00100000:2.0:1787758240.855475:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655085440:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758240.855478:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758240.855488:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758240.855498:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787758240.855507:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758240.855522:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655085568/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758240.855529:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758240/real 1787758240] req@ffff8c05a9f79500 x1874588655085440/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758240.855585:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758240.855596:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787758240.855597:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655085440/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758240.855605:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758240.855628:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11297 00000100:00080000:2.0:1787758240.855637:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758240.855640:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758240.855646:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787758240.855653:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758240.855654:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655085440:192.168.201.107@tcp:38:kworker.0 00000008:00100000:3.0:1787758240.855676:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758240.855681:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655085568:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758240.855727:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758240.855730:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655085696/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758240.855757:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758240/real 1787758240] req@ffff8c05a9f7bb80 x1874588655085568/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758240.855822:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655085568/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758240.855836:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758240.855841:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758240.855847:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655085568:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758240.855865:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655085696:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758240.855892:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758240.855962:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758240/real 1787758240] req@ffff8c05a9f79f80 x1874588655085696/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758240.855983:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655085696/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758240.855994:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758240.855998:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758240.856002:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655085696:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758240.982047:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758241.430204:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758242.006078:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758242.454151:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758243.030042:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758243.478290:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758244.054028:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758244.502072:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758245.078050:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758245.526280:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758245.974021:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758245.974032:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758245.974036:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758245.974046:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11302 00000100:00080000:3.0:1787758245.974058:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758245.974066:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758245.974108:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655085824/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758245.974155:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758245.974159:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758245.974161:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758245.974165:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11302 00000100:00080000:3.0:1787758245.974169:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758245.974172:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758245.974182:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758245.974195:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655085952/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758245.974205:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758245.974208:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758245.974210:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758245.974213:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11302 00000100:00080000:3.0:1787758245.974215:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758245.974218:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758245.974222:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758245.974230:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655086080/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758245.974239:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758245.974242:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758245.974244:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758245.974249:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11302 00000100:00080000:3.0:1787758245.974253:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758245.974257:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758245.974261:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758245.974270:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655086208/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758245.974280:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787758245.974284:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655085824:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758245.974285:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758245.974290:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758245.974300:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11302 00000100:00080000:3.0:1787758245.974304:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758245.974307:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758245.974311:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758245.974324:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655086336/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758245.974333:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758245.974336:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758245.974338:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758245.974342:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11302 00000100:00080000:3.0:1787758245.974385:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758245.974395:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758245.974401:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787758245.974407:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758245.974421:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655086464/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758245.974440:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758245.974445:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758245.974448:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758245.974455:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11302 00000100:00080000:3.0:1787758245.974461:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758245.974468:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758245.974477:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758245.974488:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758245/real 1787758245] req@ffff8c05a9f7b480 x1874588655085824/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758245.974498:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655086592/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758245.974536:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758245.974539:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758245.974541:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758245.974549:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11302 00000100:00100000:2.0:1787758245.974550:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655085824/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758245.974553:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758245.974562:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758245.974571:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787758245.974576:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758245.974584:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758245.974589:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655086720/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758245.974596:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655085824:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787758245.974618:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655085952:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758245.974652:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758245.974667:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758245/real 1787758245] req@ffff8c05a9f78380 x1874588655085952/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758245.974679:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655085952/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758245.974687:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758245.974690:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758245.974693:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655085952:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758245.974698:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655086080:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758245.974713:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758245.974722:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758245/real 1787758245] req@ffff8c05a9f78a80 x1874588655086080/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758245.974732:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655086080/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758245.974739:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758245.974741:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758245.974744:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655086080:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758245.974769:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655086208:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758245.974785:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758245.974793:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758245/real 1787758245] req@ffff8c05a9f7a680 x1874588655086208/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758245.974804:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655086208/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758245.974812:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758245.974815:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758245.974818:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655086208:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758245.974823:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655086336:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758245.974836:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758245.974845:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758245/real 1787758245] req@ffff8c05a9f7aa00 x1874588655086336/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758245.974855:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655086336/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758245.974863:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758245.974865:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758245.974868:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655086336:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758245.974872:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655086464:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758245.974926:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758245.974937:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758245/real 1787758245] req@ffff8c05a9f7a300 x1874588655086464/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758245.974948:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655086464/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758245.974955:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758245.974958:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758245.974962:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655086464:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758245.974967:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655086592:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758245.974983:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758245.974992:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758245/real 1787758245] req@ffff8c05a9f79880 x1874588655086592/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758245.975002:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655086592/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758245.975010:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758245.975013:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758245.975016:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655086592:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758245.975020:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655086720:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758245.975034:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758245.975042:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758245/real 1787758245] req@ffff8c05a9f78700 x1874588655086720/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758245.975052:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655086720/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758245.975060:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758245.975062:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758245.975065:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655086720:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758246.101950:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758246.550069:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758247.125950:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758247.574197:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758248.149937:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758248.598131:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758249.174004:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758249.622375:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758250.198013:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758250.646026:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758251.094079:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758251.094091:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758251.094096:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758251.094106:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11307 00000100:00080000:1.0:1787758251.094118:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758251.094126:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758251.094166:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655086848/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758251.094198:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758251.094204:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758251.094207:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758251.094213:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11307 00000100:00080000:1.0:1787758251.094219:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758251.094225:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758251.094252:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758251.094256:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655086848:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758251.094289:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655086976/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758251.094305:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758251.094310:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758251.094313:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758251.094318:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11307 00000100:00080000:1.0:1787758251.094322:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758251.094328:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758251.094334:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758251.094350:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655087104/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787758251.094354:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758251.094365:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758251.094370:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758251.094373:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758251.094378:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11307 00000100:00080000:1.0:1787758251.094382:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758251.094388:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758251.094394:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758251.094408:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655087232/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758251.094409:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758251/real 1787758251] req@ffff8c05b2d9b480 x1874588655086848/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758251.094424:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758251.094429:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758251.094431:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787758251.094436:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655086848/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758251.094437:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11307 00000100:00080000:1.0:1787758251.094442:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758251.094447:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:2.0:1787758251.094451:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787758251.094453:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787758251.094457:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758251.094464:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655086848:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758251.094467:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655087360/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758251.094479:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758251.094484:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758251.094486:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758251.094491:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11307 00000100:00100000:2.0:1787758251.094495:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655086976:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758251.094497:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758251.094503:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758251.094509:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758251.094524:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655087488/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787758251.094528:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758251.094539:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758251.094544:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787758251.094546:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758251/real 1787758251] req@ffff8c05b2d98000 x1874588655086976/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758251.094547:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758251.094552:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11307 00000100:00080000:1.0:1787758251.094557:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758251.094563:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758251.094567:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655086976/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787758251.094570:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787758251.094581:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787758251.094585:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655087616/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758251.094586:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758251.094592:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655086976:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758251.094600:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787758251.094602:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655087104:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758251.094605:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758251.094608:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758251.094613:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11307 00000100:00080000:1.0:1787758251.094618:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758251.094623:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:2.0:1787758251.094627:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:1.0:1787758251.094630:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758251.094644:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758251/real 1787758251] req@ffff8c05b2d99f80 x1874588655087104/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758251.094645:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655087744/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758251.094665:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655087104/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758251.094689:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758251.094694:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758251.094700:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655087104:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758251.094724:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655087232:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758251.094750:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758251.094763:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758251/real 1787758251] req@ffff8c05b2d99880 x1874588655087232/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758251.094775:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655087232/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758251.094784:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758251.094787:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758251.094790:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655087232:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758251.094795:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655087360:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758251.094811:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758251.094821:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758251/real 1787758251] req@ffff8c05b2d9b100 x1874588655087360/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758251.094832:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655087360/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758251.094840:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758251.094842:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758251.094846:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655087360:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758251.094864:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655087488:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758251.094915:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758251.094928:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758251/real 1787758251] req@ffff8c05b2d98700 x1874588655087488/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758251.094940:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655087488/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758251.094948:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758251.094951:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758251.094954:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655087488:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758251.094960:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655087616:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758251.094991:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758251.095001:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758251/real 1787758251] req@ffff8c05b2d9bb80 x1874588655087616/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758251.095013:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655087616/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758251.095021:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758251.095024:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758251.095027:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655087616:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758251.095032:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655087744:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758251.095046:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758251.095056:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758251/real 1787758251] req@ffff8c05b2d9aa00 x1874588655087744/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758251.095067:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655087744/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758251.095075:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758251.095077:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758251.095080:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655087744:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758251.222069:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758251.670203:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758252.245983:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758252.694030:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758253.270025:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758253.718136:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758254.293976:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758254.742170:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758255.317988:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758255.766247:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758256.214043:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758256.214053:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758256.214057:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758256.214067:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11312 00000100:00080000:3.0:1787758256.214079:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758256.214097:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758256.214138:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655087872/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758256.214168:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758256.214191:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758256.214193:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758256.214198:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11312 00000100:00080000:3.0:1787758256.214212:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758256.214215:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758256.214225:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758256.214241:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655088000/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758256.214252:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758256.214257:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758256.214260:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787758256.214267:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655087872:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758256.214267:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11312 00000100:00080000:3.0:1787758256.214272:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758256.214278:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758256.214286:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758256.214296:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655088128/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758256.214305:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758256.214309:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758256.214311:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758256.214315:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11312 00000100:00080000:3.0:1787758256.214318:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758256.214321:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758256.214327:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758256.214337:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655088256/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758256.214345:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758256.214349:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758256.214350:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758256.214355:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11312 00000800:00000100:2.0:1787758256.214362:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758256.214363:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758256.214370:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758256.214375:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758256.214388:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655088384/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758256.214397:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758256.214400:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758256.214402:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758256.214406:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11312 00000100:00080000:3.0:1787758256.214409:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758256.214412:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758256.214416:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758256.214425:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655088512/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758256.214433:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787758256.214440:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758256/real 1787758256] req@ffff8c05a9f78700 x1874588655087872/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758256.214440:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758256.214442:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758256.214448:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11312 00000100:00080000:3.0:1787758256.214453:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758256.214464:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758256.214471:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758256.214482:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655087872/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758256.214484:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655088640/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758256.214515:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758256.214519:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758256.214524:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787758256.214541:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787758256.214544:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655087872:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758256.214548:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758256.214558:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11312 00000100:00080000:3.0:1787758256.214564:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758256.214571:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758256.214575:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655088000:192.168.201.107@tcp:38:kworker.0 00000008:00100000:3.0:1787758256.214582:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758256.214616:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655088768/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787758256.214618:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758256.214636:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758256/real 1787758256] req@ffff8c05a9f79880 x1874588655088000/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758256.214651:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655088000/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758256.214665:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758256.214669:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758256.214673:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655088000:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758256.214681:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655088128:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758256.214696:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758256.214705:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758256/real 1787758256] req@ffff8c05a9f7a300 x1874588655088128/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758256.214715:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655088128/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758256.214722:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758256.214724:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758256.214727:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655088128:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758256.214732:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655088256:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758256.214746:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758256.214754:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758256/real 1787758256] req@ffff8c05a9f7aa00 x1874588655088256/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758256.214764:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655088256/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758256.214770:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758256.214773:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758256.214776:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655088256:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758256.214781:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655088384:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758256.214794:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758256.214802:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758256/real 1787758256] req@ffff8c05a9f7a680 x1874588655088384/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758256.214811:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655088384/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758256.214818:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758256.214821:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758256.214823:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655088384:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758256.214827:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655088512:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758256.214840:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758256.214847:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758256/real 1787758256] req@ffff8c05a9f78a80 x1874588655088512/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758256.214856:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655088512/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758256.214863:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758256.214921:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758256.214925:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655088512:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758256.214930:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655088640:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758256.214960:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758256.214971:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758256/real 1787758256] req@ffff8c05a9f78380 x1874588655088640/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758256.214982:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655088640/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758256.214989:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758256.214992:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758256.214994:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655088640:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758256.215012:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655088768:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758256.215027:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758256.215036:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758256/real 1787758256] req@ffff8c05a9f7b480 x1874588655088768/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758256.215046:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655088768/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758256.215053:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758256.215055:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758256.215058:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655088768:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758256.341985:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758256.790027:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758257.365963:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758257.814217:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758258.390007:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758258.838110:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758259.413945:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758259.862191:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758260.437973:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758260.886030:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758261.333994:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758261.334007:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758261.334012:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758261.334025:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11317 00000100:00080000:3.0:1787758261.334037:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758261.334047:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758261.334122:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655088896/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758261.334159:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758261.334164:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758261.334167:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758261.334173:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11317 00000100:00080000:3.0:1787758261.334179:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758261.334184:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758261.334196:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758261.334214:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655089024/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758261.334229:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758261.334234:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758261.334249:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758261.334253:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11317 00000100:00080000:3.0:1787758261.334257:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758261.334268:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758261.334271:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655088896:192.168.201.107@tcp:250:kworker.0 00000008:00100000:3.0:1787758261.334275:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758261.334295:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655089152/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758261.334309:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758261.334313:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758261.334316:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758261.334320:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11317 00000100:00080000:3.0:1787758261.334325:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758261.334329:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758261.334337:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758261.334351:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655089280/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758261.334364:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758261.334369:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758261.334375:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787758261.334378:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758261.334385:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11317 00000100:00080000:3.0:1787758261.334393:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758261.334400:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758261.334406:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758261.334422:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655089408/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758261.334435:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758261.334439:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758261.334442:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787758261.334449:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758261/real 1787758261] req@ffff8c05a9f79f80 x1874588655088896/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758261.334455:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11317 00000100:00080000:3.0:1787758261.334466:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758261.334480:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758261.334487:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758261.334496:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655088896/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758261.334505:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655089536/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758261.334542:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758261.334544:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758261.334554:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787758261.334556:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787758261.334559:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758261.334573:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11317 00000100:00100000:2.0:1787758261.334576:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655088896:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758261.334578:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758261.334589:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758261.334599:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758261.334611:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655089024:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758261.334616:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655089664/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758261.334637:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758261.334640:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758261.334643:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758261.334647:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11317 00000100:00080000:3.0:1787758261.334651:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758261.334656:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758261.334662:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758261.334677:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655089792/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787758261.334687:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758261.334728:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758261/real 1787758261] req@ffff8c05a9f7bb80 x1874588655089024/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758261.334751:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655089024/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758261.334763:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758261.334767:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758261.334772:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655089024:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758261.334802:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655089152:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758261.334829:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758261.334843:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758261/real 1787758261] req@ffff8c05a9f79500 x1874588655089152/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758261.334857:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655089152/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758261.334867:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758261.334871:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758261.334919:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655089152:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758261.334929:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655089280:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758261.334955:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758261.334970:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758261/real 1787758261] req@ffff8c05a9f7b800 x1874588655089280/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758261.334986:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655089280/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758261.334996:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758261.335000:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758261.335004:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655089280:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758261.335010:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655089408:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758261.335031:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758261.335042:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758261/real 1787758261] req@ffff8c05a9f78e00 x1874588655089408/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758261.335056:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655089408/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758261.335066:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758261.335069:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758261.335073:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655089408:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758261.335105:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655089536:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758261.335151:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758261.335176:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758261/real 1787758261] req@ffff8c05a9f7ad80 x1874588655089536/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758261.335201:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655089536/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758261.335218:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758261.335224:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758261.335230:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655089536:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758261.335243:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655089664:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758261.335286:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758261.335309:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758261/real 1787758261] req@ffff8c05a9f7b100 x1874588655089664/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758261.335325:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655089664/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758261.335336:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758261.335340:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758261.335344:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655089664:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758261.335352:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655089792:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758261.335378:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758261.335390:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758261/real 1787758261] req@ffff8c05a9f79180 x1874588655089792/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758261.335404:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655089792/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758261.335422:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758261.335426:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758261.335429:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655089792:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758261.462018:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758261.910165:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758262.485978:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758262.934189:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758263.510000:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758263.958266:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758264.534002:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758264.982134:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758265.557925:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758266.006076:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758266.454085:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758266.454098:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787758266.454104:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758266.454116:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11322 00000100:00080000:2.0:1787758266.454129:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758266.454139:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758266.454183:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588655089920/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758266.454215:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758266.454219:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758266.454221:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758266.454225:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11322 00000100:00080000:2.0:1787758266.454229:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758266.454233:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758266.454242:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758266.454257:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588655090048/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758266.454271:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655089920:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787758266.454271:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758266.454278:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758266.454281:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758266.454285:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11322 00000100:00080000:2.0:1787758266.454289:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758266.454296:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758266.454301:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758266.454309:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588655090176/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758266.454318:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758266.454321:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758266.454323:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758266.454326:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11322 00000100:00080000:2.0:1787758266.454329:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758266.454332:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758266.454337:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758266.454347:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588655090304/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758266.454355:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758266.454358:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758266.454360:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758266.454363:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11322 00000100:00080000:2.0:1787758266.454367:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758266.454370:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758266.454374:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758266.454395:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588655090432/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758266.454403:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758266.454406:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758266.454408:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758266.454412:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11322 00000100:00080000:2.0:1787758266.454414:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758266.454418:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758266.454422:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758266.454431:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588655090560/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758266.454440:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758266.454444:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655090048:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787758266.454445:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787758266.454450:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758266.454458:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11322 00000100:00080000:2.0:1787758266.454462:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758266.454466:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758266.454481:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758266.454502:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588655090688/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758266.454503:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655090176:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787758266.454526:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758266.454531:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758266.454534:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758266.454537:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655090304:192.168.201.107@tcp:8:kworker.0 00000100:00080000:2.0:1787758266.454559:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11322 00000100:00080000:2.0:1787758266.454567:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758266.454571:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655090432:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787758266.454578:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758266.454586:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758266.454606:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655090560:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758266.454617:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588655090816/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758266.454635:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655090688:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758266.454662:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655090816:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758266.582005:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758267.030116:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758267.605966:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758268.054148:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758268.630025:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758269.078088:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787758269.526176:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.107:988 00000400:00000100:1.0:1787758269.526221:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.107@tcp at host 192.168.201.107 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787758269.526238:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787758269.526284:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787758269.526299:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:3.0:1787758269.526312:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758266/real 1787758269] req@ffff8c05b2e71180 x1874588655090048/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787758269.526321:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787758269.526336:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787758269.526348:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787758269.526364:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:3.0:1787758269.526368:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588655090048/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787758269.526378:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00080000:3.0:1787758269.526390:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758269.526398:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758269.526403:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655090048:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758269.526412:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758266/real 1787758269] req@ffff8c05b2e72680 x1874588655090176/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758269.526428:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588655090176/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758269.526437:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758269.526441:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758269.526445:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655090176:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758269.526452:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758266/real 1787758269] req@ffff8c05b2e72d80 x1874588655090304/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758269.526464:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588655090304/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758269.526470:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758269.526472:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758269.526474:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655090304:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758269.526478:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758266/real 1787758269] req@ffff8c05b2e73100 x1874588655090432/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758269.526490:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588655090432/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758269.526498:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758269.526502:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758269.526505:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655090432:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758269.526511:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758266/real 1787758269] req@ffff8c05b2e70a80 x1874588655089920/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758269.526524:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588655089920/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758269.526532:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758269.526535:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758269.526539:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655089920:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787758269.526568:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758266/real 1787758269] req@ffff8c05b2e70000 x1874588655090816/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758269.526580:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588655090816/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758269.526588:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758269.526592:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758269.526595:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655090816:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758269.526608:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758266/real 1787758269] req@ffff8c05b2e72a00 x1874588655090560/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758269.526622:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588655090560/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758269.526629:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758269.526633:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758269.526636:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655090560:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758269.526642:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758266/real 1787758269] req@ffff8c05b2e72300 x1874588655090688/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758269.526654:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588655090688/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758269.526661:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758269.526664:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758269.526667:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655090688:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758269.654030:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758270.102215:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758270.678019:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758271.126082:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758271.574122:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758271.574133:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758271.574137:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758271.574148:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11327 00000100:00080000:3.0:1787758271.574158:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758271.574166:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758271.574202:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655090944/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758271.574243:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758271.574246:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758271.574248:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758271.574252:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11327 00000100:00080000:3.0:1787758271.574255:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758271.574258:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758271.574268:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758271.574280:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655091072/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758271.574289:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758271.574293:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758271.574295:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758271.574298:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11327 00000100:00080000:3.0:1787758271.574301:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758271.574304:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758271.574318:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758271.574327:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655091200/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758271.574339:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655090944:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758271.574341:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758271.574349:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758271.574356:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758271.574363:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11327 00000100:00080000:3.0:1787758271.574368:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758271.574374:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758271.574395:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758271.574415:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655091328/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758271.574430:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787758271.574437:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758271.574439:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758271.574445:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758271.574452:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11327 00000100:00080000:3.0:1787758271.574460:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758271.574467:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758271.574475:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758271.574494:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655091456/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758271.574500:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758271/real 1787758271] req@ffff8c05a9f79180 x1874588655090944/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758271.574531:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758271.574534:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655090944/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758271.574540:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758271.574546:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758271.574552:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11327 00000100:00080000:3.0:1787758271.574562:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758271.574572:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758271.574572:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:1.0:1787758271.574586:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000008:00100000:3.0:1787758271.574586:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758271.574596:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655090944:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787758271.574610:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655091584/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758271.574614:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655091072:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758271.574646:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758271.574651:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758271.574654:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787758271.574662:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758271.574667:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11327 00000100:00080000:3.0:1787758271.574676:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758271.574682:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758271.574690:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758271.574691:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758271/real 1787758271] req@ffff8c05a9f7b100 x1874588655091072/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758271.574713:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655091712/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758271.574718:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655091072/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758271.574743:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758271.574748:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758271.574750:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758271.574757:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758271.574760:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787758271.574767:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11327 00000100:00100000:1.0:1787758271.574773:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655091072:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758271.574777:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758271.574784:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758271.574790:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758271.574810:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655091840/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758271.574835:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655091200:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758271.574872:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758271.574954:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758271/real 1787758271] req@ffff8c05a9f7ad80 x1874588655091200/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758271.574973:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655091200/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758271.574983:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758271.574988:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758271.574992:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655091200:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758271.575001:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655091328:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758271.575032:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758271.575050:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758271/real 1787758271] req@ffff8c05a9f78e00 x1874588655091328/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758271.575066:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655091328/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758271.575078:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758271.575083:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758271.575088:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655091328:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758271.575097:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655091456:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758271.575127:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758271.575143:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758271/real 1787758271] req@ffff8c05a9f7b800 x1874588655091456/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758271.575170:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655091456/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758271.575182:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758271.575186:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758271.575190:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655091456:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758271.575198:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655091584:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758271.575226:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758271.575250:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758271/real 1787758271] req@ffff8c05a9f79500 x1874588655091584/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758271.575269:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655091584/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758271.575281:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758271.575286:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758271.575290:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655091584:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758271.575299:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655091712:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758271.575328:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758271.575345:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758271/real 1787758271] req@ffff8c05a9f7bb80 x1874588655091712/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758271.575360:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655091712/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758271.575380:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758271.575386:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758271.575391:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655091712:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758271.575400:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655091840:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758271.575429:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758271.575447:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758271/real 1787758271] req@ffff8c05a9f79f80 x1874588655091840/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758271.575465:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655091840/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758271.575486:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758271.575491:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758271.575496:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655091840:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758271.702065:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758272.150149:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758272.726026:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758273.174079:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758273.750009:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758274.198252:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758274.773994:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758275.222112:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758275.798006:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758276.246396:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758276.694035:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758276.694045:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758276.694049:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758276.694059:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11333 00000100:00080000:3.0:1787758276.694070:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758276.694078:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758276.694123:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655091968/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758276.694180:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758276.694185:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758276.694188:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758276.694194:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11333 00000100:00080000:3.0:1787758276.694200:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758276.694206:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758276.694218:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758276.694238:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655092096/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758276.694251:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758276.694256:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758276.694258:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758276.694262:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11333 00000100:00080000:3.0:1787758276.694267:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758276.694271:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758276.694278:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758276.694291:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655092224/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758276.694306:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758276.694311:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758276.694318:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758276.694320:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655091968:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758276.694327:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11333 00000100:00080000:3.0:1787758276.694338:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758276.694343:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758276.694350:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758276.694365:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655092352/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758276.694378:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758276.694383:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758276.694385:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758276.694390:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11333 00000100:00080000:3.0:1787758276.694395:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758276.694399:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758276.694404:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758276.694421:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655092480/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787758276.694428:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758276.694446:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758276.694451:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758276.694454:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758276.694461:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11333 00000100:00080000:3.0:1787758276.694466:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758276.694470:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758276.694477:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758276.694495:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655092608/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758276.694510:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758276.694512:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758276/real 1787758276] req@ffff8c05a9f7b480 x1874588655091968/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758276.694515:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758276.694521:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758276.694531:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11333 00000100:00080000:3.0:1787758276.694538:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758276.694565:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758276.694573:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758276.694594:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655092736/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758276.694595:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655091968/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758276.694625:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758276.694631:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758276.694634:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758276.694635:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758276.694644:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11333 00000100:00080000:1.0:1787758276.694649:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787758276.694650:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758276.694660:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655091968:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758276.694661:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758276.694672:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758276.694678:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655092096:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758276.694686:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655092864/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787758276.694711:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758276.694742:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758276/real 1787758276] req@ffff8c05a9f78380 x1874588655092096/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758276.694755:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655092096/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758276.694763:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758276.694768:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758276.694771:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655092096:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758276.694798:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655092224:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758276.694847:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758276.694860:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758276/real 1787758276] req@ffff8c05a9f78a80 x1874588655092224/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758276.694872:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655092224/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758276.694911:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758276.694915:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758276.694919:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655092224:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758276.694927:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655092352:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758276.694952:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758276.694975:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758276/real 1787758276] req@ffff8c05a9f7a680 x1874588655092352/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758276.694988:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655092352/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758276.694997:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758276.695001:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758276.695006:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655092352:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758276.695013:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655092480:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758276.695031:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758276.695040:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758276/real 1787758276] req@ffff8c05a9f7aa00 x1874588655092480/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758276.695050:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655092480/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758276.695059:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758276.695063:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758276.695066:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655092480:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758276.695072:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655092608:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758276.695086:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758276.695093:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758276/real 1787758276] req@ffff8c05a9f7a300 x1874588655092608/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758276.695109:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655092608/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758276.695116:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758276.695118:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758276.695121:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655092608:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758276.695125:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655092736:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758276.695140:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758276.695150:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758276/real 1787758276] req@ffff8c05a9f79880 x1874588655092736/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758276.695162:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655092736/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758276.695171:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758276.695175:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758276.695179:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655092736:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758276.695185:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655092864:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758276.695201:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758276.695210:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758276/real 1787758276] req@ffff8c05a9f78700 x1874588655092864/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758276.695219:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655092864/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758276.695232:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758276.695235:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758276.695238:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655092864:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758276.821966:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758277.270007:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758277.845935:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758278.294039:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758278.869969:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758279.318111:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758279.894051:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758280.342108:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758280.918060:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758281.366059:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758281.813972:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758281.813981:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787758281.813984:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758281.813992:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11338 00000100:00080000:2.0:1787758281.814001:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758281.814007:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758281.814040:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588655092992/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758281.814062:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758281.814064:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758281.814066:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758281.814069:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11338 00000100:00080000:2.0:1787758281.814071:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758281.814073:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758281.814081:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758281.814090:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588655093120/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758281.814099:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758281.814103:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758281.814104:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758281.814107:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11338 00000100:00080000:2.0:1787758281.814109:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758281.814112:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758281.814114:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758281.814123:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588655093248/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758281.814134:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758281.814138:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758281.814142:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758281.814144:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655092992:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787758281.814148:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11338 00000100:00080000:2.0:1787758281.814158:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758281.814163:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758281.814167:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758281.814178:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588655093376/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758281.814185:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758281.814188:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758281.814189:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758281.814191:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11338 00000100:00080000:2.0:1787758281.814195:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758281.814198:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758281.814203:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758281.814216:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588655093504/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758281.814226:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758281.814230:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758281.814232:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787758281.814239:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787758281.814245:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11338 00000100:00080000:2.0:1787758281.814253:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758281.814260:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758281.814266:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758281.814279:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588655093632/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758281.814287:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758281.814289:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787758281.814291:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758281.814296:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11338 00000100:00100000:1.0:1787758281.814298:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758281/real 1787758281] req@ffff8c05b2e70e00 x1874588655092992/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758281.814306:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758281.814318:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758281.814326:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758281.814334:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588655092992/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758281.814343:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588655093760/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758281.814349:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758281.814358:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787758281.814364:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758281.814365:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655092992:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787758281.814378:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787758281.814381:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655093120:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787758281.814386:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758281.814394:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11338 00000100:00080000:2.0:1787758281.814401:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758281.814405:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:1.0:1787758281.814412:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:2.0:1787758281.814414:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758281.814425:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758281/real 1787758281] req@ffff8c05b2e70700 x1874588655093120/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758281.814430:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588655093888/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758281.814476:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588655093120/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758281.814494:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758281.814498:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758281.814501:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655093120:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758281.814524:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655093248:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758281.814539:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758281.814546:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758281/real 1787758281] req@ffff8c05b2e73480 x1874588655093248/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758281.814554:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588655093248/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758281.814560:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758281.814562:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758281.814565:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655093248:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758281.814568:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655093376:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758281.814580:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758281.814586:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758281/real 1787758281] req@ffff8c05b2e71500 x1874588655093376/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758281.814593:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588655093376/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758281.814599:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758281.814600:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758281.814602:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655093376:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758281.814606:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655093504:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758281.814617:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758281.814623:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758281/real 1787758281] req@ffff8c05b2e73b80 x1874588655093504/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758281.814630:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588655093504/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758281.814635:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758281.814637:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758281.814639:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655093504:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758281.814642:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655093632:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758281.814653:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758281.814659:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758281/real 1787758281] req@ffff8c05b2e71c00 x1874588655093632/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758281.814666:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588655093632/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758281.814671:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758281.814673:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758281.814675:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655093632:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758281.814678:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655093760:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758281.814688:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758281.814693:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758281/real 1787758281] req@ffff8c05b2e71880 x1874588655093760/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758281.814706:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588655093760/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758281.814711:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758281.814713:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758281.814715:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655093760:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758281.814718:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655093888:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758281.814747:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758281.814755:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758281/real 1787758281] req@ffff8c05b2e70380 x1874588655093888/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758281.814763:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588655093888/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758281.814769:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758281.814771:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758281.814773:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655093888:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758281.941977:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758282.390175:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758282.965977:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758283.414114:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758283.989998:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758284.438282:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758285.013988:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758285.462107:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758286.037959:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758286.486230:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758286.934010:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758286.934024:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758286.934030:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758286.934042:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11343 00000100:00080000:3.0:1787758286.934054:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758286.934064:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758286.934108:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655094016/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758286.934145:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758286.934150:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758286.934152:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758286.934159:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11343 00000100:00080000:3.0:1787758286.934164:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758286.934169:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758286.934181:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758286.934200:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655094144/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758286.934215:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758286.934220:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758286.934223:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758286.934229:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11343 00000100:00080000:3.0:1787758286.934234:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758286.934244:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655094016:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758286.934255:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758286.934266:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758286.934288:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655094272/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758286.934304:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758286.934308:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758286.934311:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758286.934316:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11343 00000100:00080000:3.0:1787758286.934322:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758286.934329:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:1.0:1787758286.934335:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:3.0:1787758286.934337:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758286.934358:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655094400/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758286.934380:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758286.934385:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758286.934388:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758286.934396:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11343 00000100:00100000:1.0:1787758286.934398:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758286/real 1787758286] req@ffff8c05a9f78700 x1874588655094016/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758286.934402:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758286.934412:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758286.934426:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758286.934449:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655094528/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758286.934460:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655094016/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758286.934477:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758286.934490:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758286.934493:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758286.934494:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758286.934500:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758286.934505:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655094016:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758286.934519:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11343 00000100:00100000:1.0:1787758286.934521:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655094144:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758286.934529:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758286.934538:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758286.934546:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787758286.934560:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758286.934568:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655094656/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758286.934585:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758286/real 1787758286] req@ffff8c05a9f79880 x1874588655094144/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758286.934586:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758286.934598:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758286.934603:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758286.934611:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11343 00000100:00080000:3.0:1787758286.934616:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758286.934622:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758286.934655:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758286.934660:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655094144/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758286.934683:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787758286.934689:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655094784/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758286.934691:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758286.934700:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655094144:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758286.934720:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758286.934725:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758286.934728:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758286.934735:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11343 00000100:00100000:1.0:1787758286.934739:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655094272:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758286.934743:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758286.934752:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758286.934760:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787758286.934773:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758286.934781:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655094912/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758286.934789:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758286/real 1787758286] req@ffff8c05a9f7a300 x1874588655094272/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758286.934815:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655094272/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758286.934830:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758286.934834:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758286.934838:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655094272:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758286.934844:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655094400:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758286.934860:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758286.934868:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758286/real 1787758286] req@ffff8c05a9f7aa00 x1874588655094400/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758286.934913:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655094400/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758286.934926:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758286.934929:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758286.934932:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655094400:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758286.934938:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655094528:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758286.934956:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758286.934966:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758286/real 1787758286] req@ffff8c05a9f7a680 x1874588655094528/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758286.934976:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655094528/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758286.934983:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758286.934985:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758286.934989:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655094528:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758286.934994:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655094656:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758286.935008:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758286.935016:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758286/real 1787758286] req@ffff8c05a9f78a80 x1874588655094656/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758286.935025:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655094656/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758286.935032:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758286.935035:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758286.935037:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655094656:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758286.935042:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655094784:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758286.935055:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758286.935063:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758286/real 1787758286] req@ffff8c05a9f78380 x1874588655094784/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758286.935073:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655094784/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758286.935079:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758286.935082:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758286.935084:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655094784:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758286.935099:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655094912:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758286.935112:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758286.935121:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758286/real 1787758286] req@ffff8c05a9f7b480 x1874588655094912/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758286.935131:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655094912/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758286.935138:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758286.935140:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758286.935143:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655094912:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758287.062014:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758287.510091:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758288.086022:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758288.534273:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758289.109993:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758289.558105:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758290.133952:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758290.582224:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758291.157997:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758291.606078:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758292.054046:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758292.054057:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758292.054061:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758292.054072:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11348 00000100:00080000:1.0:1787758292.054084:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758292.054091:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758292.054131:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655095040/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758292.054166:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758292.054172:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758292.054174:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758292.054181:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11348 00000100:00080000:1.0:1787758292.054187:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758292.054195:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758292.054210:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758292.054216:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655095040:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758292.054237:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588655095168/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758292.054253:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758292.054257:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758292.054259:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758292.054263:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11348 00000100:00080000:1.0:1787758292.054269:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758292.054273:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758292.054279:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758292.054293:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655095296/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758292.054302:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758292.054306:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758292.054307:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758292.054310:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11348 00000100:00080000:1.0:1787758292.054313:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758292.054316:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758292.054321:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758292.054329:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655095424/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758292.054338:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758292.054341:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758292.054343:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758292.054347:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11348 00000100:00080000:1.0:1787758292.054349:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758292.054353:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758292.054357:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787758292.054362:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758292.054370:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655095552/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758292.054379:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758292.054382:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758292.054384:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758292.054388:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11348 00000100:00080000:1.0:1787758292.054392:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758292.054396:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758292.054400:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758292.054410:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655095680/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758292.054419:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758292.054422:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758292.054424:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758292.054427:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11348 00000100:00080000:1.0:1787758292.054431:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787758292.054436:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758292/real 1787758292] req@ffff8c05b2d98380 x1874588655095040/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758292.054439:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758292.054449:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758292.054478:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655095808/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787758292.054481:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655095040/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758292.054508:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758292.054514:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758292.054515:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758292.054521:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758292.054526:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787758292.054532:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11348 00000100:00100000:3.0:1787758292.054538:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655095040:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787758292.054614:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655095168:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758292.054650:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758292.054668:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787758292.054669:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758292/real 1787758292] req@ffff8c05b2d9a300 x1874588655095168/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758292.054685:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758292.054704:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758292.054709:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588655095168/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758292.054727:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787758292.054731:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655095936/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758292.054738:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758292.054754:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655095168:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758292.054775:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655095296:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758292.054801:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758292.054819:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758292/real 1787758292] req@ffff8c05b2d9aa00 x1874588655095296/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758292.054834:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655095296/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758292.054852:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758292.054856:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758292.054861:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655095296:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758292.054867:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655095424:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758292.054903:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758292.054941:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758292/real 1787758292] req@ffff8c05b2d9bb80 x1874588655095424/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758292.054953:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655095424/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758292.054961:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758292.054964:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758292.054967:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655095424:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758292.054972:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655095552:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758292.054991:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758292.055000:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758292/real 1787758292] req@ffff8c05b2d98700 x1874588655095552/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758292.055011:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655095552/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758292.055018:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758292.055021:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758292.055024:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655095552:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758292.055029:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655095680:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758292.055045:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758292.055053:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758292/real 1787758292] req@ffff8c05b2d9b100 x1874588655095680/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758292.055063:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655095680/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758292.055070:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758292.055072:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758292.055075:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655095680:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758292.055080:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655095808:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758292.055093:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758292.055101:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758292/real 1787758292] req@ffff8c05b2d99880 x1874588655095808/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758292.055110:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655095808/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758292.055118:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758292.055120:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758292.055123:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655095808:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758292.055142:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655095936:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758292.055157:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758292.055167:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758292/real 1787758292] req@ffff8c05b2d99f80 x1874588655095936/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758292.055177:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655095936/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758292.055184:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758292.055187:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758292.055190:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655095936:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758292.181958:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758292.630085:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758293.206031:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758293.654128:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758294.229958:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758294.678058:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758295.253982:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758295.702132:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758296.277974:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758296.726102:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758297.173957:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758297.173966:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758297.173969:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758297.173977:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11353 00000100:00080000:1.0:1787758297.173986:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758297.173992:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758297.174024:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655096064/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758297.174048:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758297.174050:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758297.174052:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758297.174055:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11353 00000100:00080000:1.0:1787758297.174057:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758297.174060:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758297.174078:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758297.174086:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655096192/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758297.174093:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758297.174095:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758297.174096:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758297.174099:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11353 00000100:00080000:1.0:1787758297.174100:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758297.174102:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758297.174105:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758297.174121:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655096320/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758297.174132:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787758297.174137:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655096064:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758297.174138:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758297.174143:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758297.174150:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11353 00000100:00080000:1.0:1787758297.174154:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758297.174158:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758297.174164:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758297.174179:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655096448/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758297.174190:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758297.174193:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758297.174195:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758297.174200:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11353 00000100:00080000:1.0:1787758297.174205:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787758297.174207:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758297.174214:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758297.174221:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758297.174237:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655096576/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758297.174248:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787758297.174249:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758297/real 1787758297] req@ffff8c05b2d98000 x1874588655096064/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758297.174257:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758297.174262:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758297.174268:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11353 00000100:00080000:1.0:1787758297.174272:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758297.174275:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758297.174279:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758297.174284:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655096064/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758297.174291:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655096704/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758297.174295:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758297.174304:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758297.174304:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787758297.174310:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758297.174312:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787758297.174312:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655096064:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758297.174322:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11353 00000100:00080000:1.0:1787758297.174324:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787758297.174326:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655096192:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758297.174330:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758297.174337:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758297.174347:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655096832/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758297.174354:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758297.174356:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:3.0:1787758297.174359:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758297.174360:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758297.174378:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11353 00000100:00080000:1.0:1787758297.174381:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787758297.174386:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758297/real 1787758297] req@ffff8c05b2d9b480 x1874588655096192/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758297.174387:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758297.174393:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758297.174407:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655096960/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787758297.174408:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655096192/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758297.174434:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758297.174438:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758297.174441:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655096192:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758297.174482:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655096320:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758297.174498:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758297.174533:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758297/real 1787758297] req@ffff8c05b2d99f80 x1874588655096320/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758297.174544:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655096320/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758297.174552:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758297.174555:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758297.174557:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655096320:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758297.174562:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655096448:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758297.174578:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758297.174586:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758297/real 1787758297] req@ffff8c05b2d99880 x1874588655096448/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758297.174603:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655096448/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758297.174610:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758297.174612:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758297.174614:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655096448:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758297.174619:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655096576:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758297.174650:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758297.174660:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758297/real 1787758297] req@ffff8c05b2d9b100 x1874588655096576/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758297.174672:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655096576/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758297.174680:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758297.174684:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758297.174687:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655096576:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758297.174693:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655096704:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758297.174715:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758297.174726:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758297/real 1787758297] req@ffff8c05b2d98700 x1874588655096704/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758297.174737:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655096704/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758297.174747:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758297.174749:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758297.174751:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655096704:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758297.174755:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655096832:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758297.174769:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758297.174776:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758297/real 1787758297] req@ffff8c05b2d9bb80 x1874588655096832/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758297.174783:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655096832/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758297.174789:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758297.174791:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758297.174793:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655096832:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758297.174796:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655096960:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758297.174805:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758297.174811:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758297/real 1787758297] req@ffff8c05b2d9aa00 x1874588655096960/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758297.174818:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655096960/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758297.174823:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758297.174824:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758297.174827:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655096960:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758297.302112:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758297.750098:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758298.325982:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758298.774286:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758299.349998:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758299.798119:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758300.374014:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758300.822176:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758301.397969:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758301.846125:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758302.294030:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758302.294042:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758302.294047:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758302.294057:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11358 00000100:00080000:3.0:1787758302.294068:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758302.294075:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758302.294113:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655097088/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758302.294142:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758302.294146:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758302.294148:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758302.294152:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11358 00000100:00080000:3.0:1787758302.294155:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758302.294158:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758302.294167:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758302.294180:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655097216/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758302.294191:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758302.294195:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655097088:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758302.294196:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758302.294199:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758302.294209:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11358 00000100:00080000:3.0:1787758302.294213:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758302.294216:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758302.294220:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758302.294232:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655097344/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758302.294241:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758302.294245:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758302.294247:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758302.294250:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11358 00000100:00080000:3.0:1787758302.294253:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758302.294256:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758302.294260:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758302.294269:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655097472/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758302.294278:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758302.294282:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758302.294285:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758302.294290:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11358 00000100:00080000:3.0:1787758302.294295:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787758302.294302:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758302.294309:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758302.294319:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758302.294338:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655097600/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758302.294354:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758302/real 1787758302] req@ffff8c05a9f79f80 x1874588655097088/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758302.294388:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758302.294395:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758302.294398:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758302.294406:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655097088/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758302.294410:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11358 00000100:00080000:3.0:1787758302.294421:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758302.294430:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758302.294442:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787758302.294457:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787758302.294462:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655097728/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758302.294465:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758302.294479:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655097088:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758302.294482:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758302.294491:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758302.294494:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758302.294499:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11358 00000100:00080000:3.0:1787758302.294503:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758302.294507:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758302.294510:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758302.294519:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655097856/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758302.294529:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655097216:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758302.294532:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758302.294538:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758302.294540:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758302.294544:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11358 00000100:00080000:3.0:1787758302.294548:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758302.294552:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758302.294557:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787758302.294571:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758302.294572:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655097984/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758302.294593:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758302/real 1787758302] req@ffff8c05a9f7bb80 x1874588655097216/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758302.294608:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655097216/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758302.294617:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758302.294620:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758302.294623:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655097216:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758302.294631:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655097344:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758302.294658:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758302.294674:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758302/real 1787758302] req@ffff8c05a9f79500 x1874588655097344/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758302.294692:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655097344/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758302.294704:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758302.294710:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758302.294713:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655097344:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758302.294720:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655097472:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758302.294744:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758302.294755:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758302/real 1787758302] req@ffff8c05a9f7b800 x1874588655097472/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758302.294765:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655097472/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758302.294773:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758302.294776:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758302.294779:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655097472:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758302.294784:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655097600:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758302.294798:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758302.294819:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758302/real 1787758302] req@ffff8c05a9f78e00 x1874588655097600/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758302.294847:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655097600/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758302.294854:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758302.294857:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758302.294860:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655097600:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758302.294865:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655097728:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758302.294883:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758302.294917:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758302/real 1787758302] req@ffff8c05a9f7ad80 x1874588655097728/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758302.294930:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655097728/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758302.294938:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758302.294941:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758302.294944:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655097728:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758302.294964:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655097856:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758302.294995:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758302.295007:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758302/real 1787758302] req@ffff8c05a9f7b100 x1874588655097856/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758302.295019:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655097856/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758302.295026:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758302.295030:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758302.295033:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655097856:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758302.295038:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655097984:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758302.295053:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758302.295062:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758302/real 1787758302] req@ffff8c05a9f79180 x1874588655097984/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758302.295079:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655097984/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758302.295099:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758302.295105:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758302.295110:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655097984:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758302.421963:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758302.870102:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758303.446034:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758303.894141:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758304.470001:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758304.918123:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758305.493956:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758305.942125:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758306.517994:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758306.966258:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758307.414058:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758307.414072:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787758307.414077:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758307.414090:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11363 00000100:00080000:2.0:1787758307.414105:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758307.414115:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758307.414162:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73800 x1874588655098112/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758307.414201:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758307.414206:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758307.414209:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758307.414214:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11363 00000100:00080000:2.0:1787758307.414219:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758307.414223:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758307.414235:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758307.414255:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588655098240/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758307.414313:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758307.414321:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655098112:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787758307.414322:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758307.414328:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758307.414341:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11363 00000100:00080000:2.0:1787758307.414346:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758307.414351:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758307.414358:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758307.414377:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588655098368/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758307.414391:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758307.414395:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758307.414397:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758307.414402:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11363 00000100:00080000:2.0:1787758307.414407:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758307.414411:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758307.414418:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787758307.414424:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758307.414439:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588655098496/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758307.414454:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758307.414458:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758307.414460:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758307.414464:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11363 00000100:00080000:2.0:1787758307.414468:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758307.414473:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758307.414479:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758307.414489:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758307/real 1787758307] req@ffff8c05b2e73800 x1874588655098112/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758307.414498:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588655098624/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758307.414543:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758307.414548:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758307.414550:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758307.414558:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73800 x1874588655098112/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758307.414559:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11363 00000100:00080000:2.0:1787758307.414568:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758307.414578:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758307.414589:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758307.414608:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588655098752/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758307.414610:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758307.414632:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787758307.414633:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758307.414643:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787758307.414645:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758307.414651:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655098112:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787758307.414654:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11363 00000100:00080000:2.0:1787758307.414664:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758307.414671:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758307.414676:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758307.414687:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588655098880/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758307.414697:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758307.414700:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758307.414702:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758307.414706:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11363 00000100:00080000:2.0:1787758307.414711:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758307.414721:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758307.414722:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655098240:192.168.201.107@tcp:38:kworker.0 00000008:00100000:2.0:1787758307.414728:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758307.414744:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588655099008/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787758307.414774:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758307.414800:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758307/real 1787758307] req@ffff8c05b2e70380 x1874588655098240/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758307.414816:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588655098240/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758307.414834:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758307.414838:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758307.414842:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655098240:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758307.414850:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655098368:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758307.414870:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758307.414925:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758307/real 1787758307] req@ffff8c05b2e71880 x1874588655098368/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758307.414952:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588655098368/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758307.414962:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758307.414966:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758307.414970:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655098368:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758307.414977:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655098496:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758307.415003:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758307.415015:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758307/real 1787758307] req@ffff8c05b2e71c00 x1874588655098496/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758307.415029:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588655098496/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758307.415038:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758307.415041:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758307.415045:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655098496:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758307.415052:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655098624:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758307.415070:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758307.415081:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758307/real 1787758307] req@ffff8c05b2e73b80 x1874588655098624/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758307.415093:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588655098624/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758307.415102:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758307.415105:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758307.415109:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655098624:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758307.415115:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655098752:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758307.415132:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758307.415143:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758307/real 1787758307] req@ffff8c05b2e71500 x1874588655098752/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758307.415155:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588655098752/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758307.415164:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758307.415167:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758307.415171:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655098752:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758307.415176:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655098880:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758307.415193:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758307.415204:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758307/real 1787758307] req@ffff8c05b2e73480 x1874588655098880/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758307.415217:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588655098880/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758307.415225:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758307.415245:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758307.415250:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655098880:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758307.415322:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655099008:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758307.415349:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758307.415362:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758307/real 1787758307] req@ffff8c05b2e70700 x1874588655099008/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758307.415376:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588655099008/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758307.415386:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758307.415390:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758307.415394:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655099008:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758307.542002:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758307.990106:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758308.565994:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758309.014223:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758309.590008:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758310.038207:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758310.613998:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758311.062261:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758311.638022:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758312.086114:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758312.534270:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758312.534283:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758312.534288:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758312.534300:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11368 00000100:00080000:3.0:1787758312.534313:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758312.534322:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758312.534375:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655099136/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758312.534414:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758312.534419:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758312.534423:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758312.534432:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11368 00000100:00080000:3.0:1787758312.534439:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758312.534446:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758312.534464:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758312.534494:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655099264/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758312.534501:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655099136:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758312.534535:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758312.534542:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758312.534546:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758312.534565:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11368 00000100:00080000:3.0:1787758312.534574:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758312.534582:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758312.534594:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787758312.534600:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758312.534632:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655099392/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758312.534656:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758312.534665:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758312/real 1787758312] req@ffff8c05a9f79180 x1874588655099136/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758312.534667:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758312.534674:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758312.534692:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11368 00000100:00080000:3.0:1787758312.534706:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758312.534719:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758312.534730:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655099136/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787758312.534735:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787758312.534765:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787758312.534772:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655099520/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758312.534779:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758312.534792:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655099136:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758312.534821:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758312.534829:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758312.534834:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758312.534839:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655099264:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758312.534849:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11368 00000100:00080000:3.0:1787758312.534860:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758312.534867:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758312.534924:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787758312.534934:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758312.534956:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655099648/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758312.534974:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758312/real 1787758312] req@ffff8c05a9f7b100 x1874588655099264/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758312.534979:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758312.534985:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758312.534988:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758312.534994:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11368 00000100:00080000:3.0:1787758312.534999:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758312.535008:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758312.535018:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758312.535038:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655099776/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758312.535054:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655099264/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758312.535055:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758312.535081:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758312.535085:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758312.535105:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11368 00000100:00080000:1.0:1787758312.535113:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758312.535116:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758312.535132:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787758312.535135:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758312.535150:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655099264:192.168.201.107@tcp:38:kworker.0 00000008:00100000:3.0:1787758312.535152:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758312.535178:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655099392:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758312.535229:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655099904/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758312.535264:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758312.535271:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1787758312.535282:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758312.535282:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758312.535298:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11368 00000100:00080000:3.0:1787758312.535313:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758312.535323:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758312.535331:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758312/real 1787758312] req@ffff8c05a9f7ad80 x1874588655099392/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787758312.535338:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758312.535370:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655100032/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758312.535375:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655099392/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758312.535424:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758312.535428:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758312.535435:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655099392:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758312.535444:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655099520:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758312.535478:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758312.535494:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758312/real 1787758312] req@ffff8c05a9f78e00 x1874588655099520/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758312.535512:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655099520/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758312.535532:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758312.535538:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758312.535545:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655099520:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758312.535577:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655099648:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758312.535616:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758312.535632:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758312/real 1787758312] req@ffff8c05a9f7b800 x1874588655099648/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758312.535659:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655099648/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758312.535677:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758312.535686:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758312.535695:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655099648:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758312.535709:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655099776:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758312.535754:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758312.535777:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758312/real 1787758312] req@ffff8c05a9f79500 x1874588655099776/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758312.535798:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655099776/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758312.535810:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758312.535817:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758312.535824:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655099776:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758312.535837:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655099904:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758312.535910:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758312.535936:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758312/real 1787758312] req@ffff8c05a9f7bb80 x1874588655099904/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758312.535963:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655099904/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758312.535984:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758312.535995:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758312.536003:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655099904:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758312.536016:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655100032:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758312.536057:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758312.536084:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758312/real 1787758312] req@ffff8c05a9f79f80 x1874588655100032/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758312.536113:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655100032/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758312.536131:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758312.536140:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758312.536150:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655100032:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758312.662041:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758313.110276:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758313.686002:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758314.134103:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758314.709952:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758315.158109:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758315.734003:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758316.182181:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758316.757980:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758317.206034:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758317.654038:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758317.654054:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758317.654060:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758317.654076:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11374 00000100:00080000:3.0:1787758317.654145:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758317.654160:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758317.654210:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655100160/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758317.654250:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758317.654255:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758317.654258:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758317.654263:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11374 00000100:00080000:3.0:1787758317.654269:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758317.654275:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758317.654286:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758317.654303:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655100288/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758317.654318:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758317.654323:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758317.654325:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758317.654335:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11374 00000100:00100000:1.0:1787758317.654340:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655100160:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758317.654340:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758317.654349:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758317.654365:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758317.654394:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655100416/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758317.654416:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758317.654423:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758317.654427:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758317.654435:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11374 00000100:00080000:3.0:1787758317.654444:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758317.654453:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:1.0:1787758317.654466:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:3.0:1787758317.654466:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758317.654495:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655100544/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758317.654518:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758317.654522:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758317.654525:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758317.654531:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11374 00000100:00080000:3.0:1787758317.654537:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758317.654546:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758317.654551:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758317.654563:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758317/real 1787758317] req@ffff8c05a9f7b480 x1874588655100160/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758317.654565:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655100672/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758317.654608:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758317.654619:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758317.654622:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758317.654629:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11374 00000100:00080000:3.0:1787758317.654635:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758317.654645:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758317.654646:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655100160/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787758317.654651:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758317.654673:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655100800/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758317.654679:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758317.654695:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758317.654700:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787758317.654702:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758317.654712:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758317.654720:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655100160:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758317.654724:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11374 00000100:00080000:3.0:1787758317.654733:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758317.654741:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758317.654747:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758317.654760:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655100928/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758317.654773:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758317.654776:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655100288:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758317.654779:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758317.654786:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758317.654798:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11374 00000100:00080000:3.0:1787758317.654803:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758317.654808:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758317.654814:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758317.654830:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655101056/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787758317.654845:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758317.654946:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758317/real 1787758317] req@ffff8c05a9f78380 x1874588655100288/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758317.654990:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655100288/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758317.655008:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758317.655016:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758317.655023:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655100288:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758317.655037:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655100416:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758317.655122:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758317.655147:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758317/real 1787758317] req@ffff8c05a9f78a80 x1874588655100416/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758317.655171:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655100416/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758317.655186:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758317.655193:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758317.655200:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655100416:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758317.655212:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655100544:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758317.655248:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758317.655266:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758317/real 1787758317] req@ffff8c05a9f7a680 x1874588655100544/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758317.655288:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655100544/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758317.655314:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758317.655320:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758317.655327:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655100544:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758317.655338:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655100672:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758317.655372:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758317.655392:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758317/real 1787758317] req@ffff8c05a9f7aa00 x1874588655100672/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758317.655413:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655100672/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758317.655428:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758317.655434:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758317.655440:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655100672:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758317.655450:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655100800:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758317.655480:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758317.655499:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758317/real 1787758317] req@ffff8c05a9f7a300 x1874588655100800/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758317.655521:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655100800/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758317.655536:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758317.655542:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758317.655549:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655100800:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758317.655584:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655100928:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758317.655619:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758317.655638:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758317/real 1787758317] req@ffff8c05a9f79880 x1874588655100928/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758317.655658:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655100928/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758317.655672:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758317.655677:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758317.655684:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655100928:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758317.655694:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655101056:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758317.655722:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758317.655741:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758317/real 1787758317] req@ffff8c05a9f78700 x1874588655101056/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758317.655762:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655101056/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758317.655776:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758317.655782:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758317.655787:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655101056:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758317.782041:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758318.230175:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758318.806007:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758319.254123:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758319.829952:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758320.278125:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758320.853981:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758321.302261:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758321.877990:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758322.326250:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758322.773999:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758322.774018:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758322.774022:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758322.774031:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11379 00000100:00080000:3.0:1787758322.774041:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758322.774047:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758322.774082:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655101184/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758322.774113:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758322.774116:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758322.774117:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758322.774121:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11379 00000100:00080000:3.0:1787758322.774123:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758322.774126:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758322.774134:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758322.774144:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655101312/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758322.774151:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758322.774154:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758322.774156:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758322.774158:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11379 00000100:00080000:3.0:1787758322.774160:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758322.774162:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758322.774165:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758322.774172:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655101440/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758322.774177:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655101184:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758322.774183:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758322.774186:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758322.774187:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758322.774189:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11379 00000100:00080000:3.0:1787758322.774192:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758322.774194:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758322.774198:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758322.774206:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655101568/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758322.774213:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758322.774216:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758322.774218:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758322.774220:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11379 00000100:00080000:3.0:1787758322.774222:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758322.774224:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758322.774227:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758322.774235:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655101696/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758322.774241:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787758322.774244:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758322.774245:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758322.774248:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758322.774252:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11379 00000100:00080000:3.0:1787758322.774254:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758322.774257:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758322.774260:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758322.774267:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655101824/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758322.774276:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758322.774278:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758322.774279:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758322.774283:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11379 00000100:00100000:1.0:1787758322.774286:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758322/real 1787758322] req@ffff8c05a9f78700 x1874588655101184/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758322.774287:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758322.774299:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758322.774303:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758322.774304:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655101184/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758322.774313:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655101952/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758322.774314:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758322.774323:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758322.774330:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655101184:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758322.774331:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758322.774337:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758322.774339:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758322.774341:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655101312:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758322.774345:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11379 00000100:00080000:3.0:1787758322.774348:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758322.774351:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758322.774354:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787758322.774365:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758322.774366:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655102080/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758322.774378:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758322/real 1787758322] req@ffff8c05a9f79880 x1874588655101312/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758322.774387:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655101312/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758322.774392:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758322.774395:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758322.774398:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655101312:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758322.774418:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655101440:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758322.774430:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758322.774437:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758322/real 1787758322] req@ffff8c05a9f7a300 x1874588655101440/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758322.774444:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655101440/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758322.774450:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758322.774452:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758322.774454:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655101440:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758322.774457:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655101568:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758322.774467:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758322.774472:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758322/real 1787758322] req@ffff8c05a9f7aa00 x1874588655101568/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758322.774479:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655101568/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758322.774484:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758322.774486:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758322.774488:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655101568:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758322.774491:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655101696:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758322.774501:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758322.774507:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758322/real 1787758322] req@ffff8c05a9f7a680 x1874588655101696/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758322.774514:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655101696/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758322.774533:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758322.774535:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758322.774538:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655101696:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758322.774541:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655101824:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758322.774553:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758322.774560:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758322/real 1787758322] req@ffff8c05a9f78a80 x1874588655101824/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758322.774567:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655101824/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758322.774572:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758322.774574:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758322.774576:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655101824:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758322.774579:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655101952:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758322.774589:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758322.774595:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758322/real 1787758322] req@ffff8c05a9f78380 x1874588655101952/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758322.774602:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655101952/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758322.774606:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758322.774608:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758322.774611:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655101952:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758322.774614:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655102080:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758322.774623:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758322.774629:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758322/real 1787758322] req@ffff8c05a9f7b480 x1874588655102080/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758322.774636:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655102080/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758322.774641:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758322.774642:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758322.774644:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655102080:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758322.901993:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758323.350318:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758323.925966:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758324.374167:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758324.950029:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758325.398270:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758325.974039:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758326.422161:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758326.998049:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758327.446201:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758327.894099:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758327.894115:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787758327.894121:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758327.894134:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11384 00000100:00080000:2.0:1787758327.894150:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758327.894161:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758327.894211:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588655102208/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758327.894266:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758327.894273:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758327.894277:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758327.894285:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11384 00000100:00080000:2.0:1787758327.894293:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758327.894301:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758327.894316:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758327.894338:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655102208:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787758327.894340:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588655102336/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758327.894372:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758327.894378:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758327.894398:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758327.894404:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11384 00000100:00080000:2.0:1787758327.894408:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758327.894414:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758327.894421:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758327.894436:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588655102464/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787758327.894441:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787758327.894452:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758327.894457:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758327.894459:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758327.894465:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11384 00000100:00080000:2.0:1787758327.894469:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758327.894475:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758327.894482:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758327.894497:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588655102592/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758327.894513:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758327.894515:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758327/real 1787758327] req@ffff8c05b2e70e00 x1874588655102208/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758327.894518:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758327.894522:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758327.894528:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11384 00000100:00080000:2.0:1787758327.894533:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758327.894539:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758327.894543:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588655102208/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787758327.894546:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787758327.894560:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787758327.894563:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588655102720/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758327.894567:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758327.894573:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655102208:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787758327.894578:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758327.894583:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758327.894586:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758327.894592:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11384 00000100:00080000:2.0:1787758327.894597:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758327.894602:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758327.894620:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758327.894635:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655102336:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758327.894636:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588655102848/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758327.894652:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758327.894657:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787758327.894660:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758327.894665:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11384 00000800:00000100:1.0:1787758327.894671:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787758327.894671:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758327.894677:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758327.894702:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758327.894711:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758327/real 1787758327] req@ffff8c05b2e72300 x1874588655102336/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758327.894721:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588655102976/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758327.894752:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758327.894753:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588655102336/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758327.894759:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758327.894763:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758327.894777:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11384 00000100:00080000:1.0:1787758327.894778:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758327.894783:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758327.894784:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787758327.894790:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758327.894792:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655102336:192.168.201.107@tcp:38:kworker.0 00000008:00100000:2.0:1787758327.894799:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758327.894803:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655102464:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758327.894817:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588655103104/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787758327.894838:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758327.894860:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758327/real 1787758327] req@ffff8c05b2e72a00 x1874588655102464/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758327.894967:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588655102464/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758327.894977:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758327.894981:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758327.894985:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655102464:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758327.894993:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655102592:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758327.895020:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758327.895033:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758327/real 1787758327] req@ffff8c05b2e70000 x1874588655102592/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758327.895046:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588655102592/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758327.895055:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758327.895058:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758327.895061:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655102592:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758327.895067:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655102720:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758327.895083:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758327.895093:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758327/real 1787758327] req@ffff8c05b2e70a80 x1874588655102720/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758327.895105:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588655102720/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758327.895113:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758327.895116:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758327.895119:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655102720:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758327.895137:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655102848:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758327.895154:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758327.895164:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758327/real 1787758327] req@ffff8c05b2e73100 x1874588655102848/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758327.895175:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588655102848/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758327.895191:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758327.895194:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758327.895198:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655102848:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758327.895203:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655102976:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758327.895218:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758327.895227:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758327/real 1787758327] req@ffff8c05b2e72d80 x1874588655102976/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758327.895257:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588655102976/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758327.895267:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758327.895270:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758327.895274:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655102976:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758327.895280:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655103104:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758327.895299:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758327.895310:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758327/real 1787758327] req@ffff8c05b2e72680 x1874588655103104/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758327.895323:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588655103104/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758327.895332:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758327.895336:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758327.895340:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655103104:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758328.021943:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758328.470018:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758329.045981:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758329.494049:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758330.069979:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758330.517989:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758331.093977:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758331.542173:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758332.117962:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758332.566084:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758333.013986:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758333.013997:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787758333.014001:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758333.014012:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11389 00000100:00080000:2.0:1787758333.014024:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758333.014031:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758333.014071:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588655103232/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758333.014104:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758333.014108:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758333.014111:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758333.014115:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11389 00000100:00080000:2.0:1787758333.014129:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758333.014133:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758333.014143:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758333.014157:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588655103360/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758333.014169:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758333.014173:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758333.014175:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758333.014190:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11389 00000100:00080000:2.0:1787758333.014193:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758333.014198:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758333.014204:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655103232:192.168.201.107@tcp:250:kworker.0 00000008:00100000:2.0:1787758333.014206:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758333.014222:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588655103488/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758333.014235:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758333.014238:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758333.014240:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758333.014243:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11389 00000100:00080000:2.0:1787758333.014245:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758333.014249:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758333.014254:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758333.014265:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588655103616/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758333.014275:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758333.014278:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758333.014280:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758333.014282:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11389 00000100:00080000:2.0:1787758333.014287:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758333.014290:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758333.014295:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758333.014304:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588655103744/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758333.014313:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758333.014316:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758333.014318:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758333.014322:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11389 00000100:00080000:2.0:1787758333.014325:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758333.014328:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758333.014333:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758333.014338:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655103360:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758333.014348:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588655103872/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758333.014359:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758333.014362:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787758333.014365:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758333.014370:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11389 00000100:00080000:2.0:1787758333.014375:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758333.014379:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758333.014385:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758333.014389:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655103488:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758333.014401:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588655104000/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758333.014415:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758333.014418:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758333.014421:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758333.014427:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11389 00000100:00100000:1.0:1787758333.014432:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655103616:192.168.201.107@tcp:8:kworker.0 00000100:00080000:2.0:1787758333.014434:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758333.014445:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758333.014451:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758333.014463:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588655104128/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758333.014472:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655103744:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758333.014505:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655103872:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758333.014529:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655104000:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758333.014551:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655104128:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758333.141979:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758333.590134:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758334.166015:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758334.614129:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758335.189960:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758335.638045:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787758336.086170:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.107:988 00000400:00000100:3.0:1787758336.086232:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.107@tcp at host 192.168.201.107 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787758336.086262:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:3.0:1787758336.086305:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:3.0:1787758336.086318:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:3.0:1787758336.086328:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:3.0:1787758336.086363:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:1.0:1787758336.086365:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758333/real 1787758336] req@ffff8c05b2e72d80 x1874588655103488/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787758336.086378:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:3.0:1787758336.086394:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:1.0:1787758336.086409:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588655103488/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787758336.086412:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00080000:1.0:1787758336.086439:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758336.086446:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758336.086452:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655103488:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758336.086461:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758333/real 1787758336] req@ffff8c05b2e73100 x1874588655103616/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758336.086476:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588655103616/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758336.086483:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758336.086487:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758336.086490:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655103616:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758336.086498:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758333/real 1787758336] req@ffff8c05b2e70a80 x1874588655103744/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758336.086509:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588655103744/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758336.086518:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758336.086520:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758336.086524:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655103744:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758336.086531:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758333/real 1787758336] req@ffff8c05b2e70000 x1874588655103872/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758336.086554:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588655103872/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758336.086563:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758336.086567:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758336.086571:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655103872:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758336.086580:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758333/real 1787758336] req@ffff8c05b2e71180 x1874588655103232/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758336.086604:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588655103232/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758336.086612:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758336.086616:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758336.086619:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655103232:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758336.086626:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758333/real 1787758336] req@ffff8c05b2e72680 x1874588655103360/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758336.086638:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588655103360/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758336.086645:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758336.086649:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758336.086652:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655103360:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758336.086694:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758333/real 1787758336] req@ffff8c05b2e72a00 x1874588655104000/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758336.086706:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588655104000/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758336.086714:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758336.086718:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758336.086723:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655104000:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758336.086730:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758333/real 1787758336] req@ffff8c05b2e72300 x1874588655104128/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758336.086758:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588655104128/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758336.086787:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758336.086792:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758336.086795:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655104128:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758336.213995:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758336.662217:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758337.237946:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758337.686061:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758338.133984:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758338.133992:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758338.133996:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758338.134004:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11394 00000100:00080000:0.0:1787758338.134014:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758338.134020:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758338.134052:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655104256/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758338.134075:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758338.134078:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758338.134079:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758338.134082:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11394 00000100:00080000:0.0:1787758338.134084:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758338.134086:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758338.134094:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758338.134109:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655104384/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758338.134119:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655104256:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758338.134120:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758338.134128:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758338.134130:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758338.134133:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11394 00000100:00080000:0.0:1787758338.134135:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758338.134139:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758338.134144:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758338.134158:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588655104512/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758338.134168:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758338.134173:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758338.134175:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758338.134178:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11394 00000100:00080000:0.0:1787758338.134182:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758338.134189:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:1.0:1787758338.134189:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:0.0:1787758338.134199:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758338.134213:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588655104640/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758338.134224:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758338.134228:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758338.134230:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758338.134235:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758338/real 1787758338] req@ffff8c05a9f76680 x1874588655104256/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758338.134238:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11394 00000100:00080000:0.0:1787758338.134246:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758338.134253:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758338.134257:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655104256/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787758338.134261:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787758338.134269:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758338.134276:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758338.134281:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588655104768/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758338.134283:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655104256:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758338.134300:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758338.134304:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758338.134307:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758338.134311:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11394 00000100:00080000:0.0:1787758338.134316:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758338.134320:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758338.134326:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758338.134327:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655104384:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758338.134343:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655104896/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758338.134356:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758338.134361:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1787758338.134364:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758338.134365:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758338.134373:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11394 00000100:00080000:0.0:1787758338.134377:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758338.134382:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758338.134387:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758338/real 1787758338] req@ffff8c05a9f77800 x1874588655104384/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787758338.134391:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758338.134411:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655105024/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758338.134413:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655104384/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758338.134440:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758338.134442:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758338.134445:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787758338.134451:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787758338.134454:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655104384:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758338.134456:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758338.134466:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11394 00000100:00100000:1.0:1787758338.134468:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655104512:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758338.134473:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758338.134481:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758338.134488:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787758338.134505:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758338.134506:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655105152/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758338.134523:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758338/real 1787758338] req@ffff8c05a9f75c00 x1874588655104512/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758338.134539:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588655104512/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758338.134547:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758338.134551:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758338.134555:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655104512:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758338.134562:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655104640:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758338.134584:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758338.134594:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758338/real 1787758338] req@ffff8c05a9f76a00 x1874588655104640/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758338.134605:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588655104640/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758338.134618:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758338.134622:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758338.134625:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655104640:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758338.134630:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655104768:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758338.134672:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758338.134685:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758338/real 1787758338] req@ffff8c05a9f77b80 x1874588655104768/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758338.134697:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588655104768/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758338.134706:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758338.134710:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758338.134714:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655104768:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758338.134747:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655104896:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758338.134770:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758338.134783:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758338/real 1787758338] req@ffff8c05a9f76300 x1874588655104896/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758338.134796:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655104896/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758338.134805:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758338.134809:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758338.134813:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655104896:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758338.134819:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655105024:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758338.134841:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758338.134854:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758338/real 1787758338] req@ffff8c05a9f74380 x1874588655105024/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758338.134865:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655105024/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758338.134873:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758338.134979:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758338.134983:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655105024:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758338.134991:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655105152:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758338.135018:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758338.135031:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758338/real 1787758338] req@ffff8c05a9f74a80 x1874588655105152/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758338.135045:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655105152/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758338.135054:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758338.135058:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758338.135061:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655105152:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758338.261972:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758338.710014:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758339.285951:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758339.734094:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758340.309959:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758340.758000:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758341.333962:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758341.782081:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758342.357945:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758342.806022:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758343.253965:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758343.253973:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758343.253976:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758343.253985:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11399 00000100:00080000:0.0:1787758343.253994:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758343.254000:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758343.254031:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588655105280/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758343.254054:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758343.254057:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758343.254058:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758343.254061:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11399 00000100:00080000:0.0:1787758343.254063:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758343.254066:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758343.254074:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758343.254083:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588655105408/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758343.254089:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758343.254092:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758343.254093:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758343.254096:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11399 00000100:00080000:0.0:1787758343.254099:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758343.254101:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758343.254104:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758343.254110:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588655105536/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758343.254117:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758343.254119:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758343.254120:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758343.254123:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11399 00000100:00080000:0.0:1787758343.254125:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758343.254129:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655105280:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758343.254130:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758343.254138:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758343.254146:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588655105664/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758343.254153:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758343.254156:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758343.254157:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758343.254160:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11399 00000100:00080000:0.0:1787758343.254162:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758343.254165:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758343.254168:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758343.254176:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588655105792/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758343.254182:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758343.254184:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758343.254185:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758343.254187:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11399 00000100:00080000:0.0:1787758343.254190:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787758343.254193:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758343.254195:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758343.254199:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758343.254207:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655105920/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758343.254213:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758343.254215:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758343.254216:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758343.254218:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11399 00000100:00080000:0.0:1787758343.254220:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758343.254223:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758343.254227:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758343.254235:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758343/real 1787758343] req@ffff8c05a9f74e00 x1874588655105280/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758343.254237:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588655106048/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758343.254270:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758343.254272:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758343.254274:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758343.254279:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588655105280/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758343.254280:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11399 00000100:00080000:0.0:1787758343.254286:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758343.254293:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758343.254299:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787758343.254300:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758343.254306:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758343.254312:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655106176/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758343.254313:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655105280:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758343.254325:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655105408:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758343.254345:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758343.254356:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758343/real 1787758343] req@ffff8c05a9f77100 x1874588655105408/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758343.254366:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588655105408/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758343.254372:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758343.254375:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758343.254377:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655105408:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758343.254380:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655105536:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758343.254391:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758343.254398:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758343/real 1787758343] req@ffff8c05a9f76d80 x1874588655105536/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758343.254417:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588655105536/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758343.254436:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758343.254444:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758343.254449:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655105536:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758343.254474:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655105664:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758343.254491:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758343.254498:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758343/real 1787758343] req@ffff8c05a9f75180 x1874588655105664/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758343.254507:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588655105664/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758343.254512:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758343.254514:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758343.254516:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655105664:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758343.254520:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655105792:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758343.254530:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758343.254535:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758343/real 1787758343] req@ffff8c05a9f74000 x1874588655105792/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758343.254543:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588655105792/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758343.254552:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758343.254554:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758343.254556:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655105792:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758343.254559:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655105920:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758343.254569:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758343.254574:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758343/real 1787758343] req@ffff8c05a9f75880 x1874588655105920/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758343.254581:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655105920/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758343.254585:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758343.254587:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758343.254589:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655105920:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758343.254593:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655106048:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758343.254602:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758343.254607:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758343/real 1787758343] req@ffff8c05a9f75500 x1874588655106048/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758343.254614:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588655106048/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758343.254619:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758343.254621:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758343.254623:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655106048:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758343.254627:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655106176:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758343.254662:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758343.254670:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758343/real 1787758343] req@ffff8c05a9f74a80 x1874588655106176/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758343.254678:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655106176/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758343.254683:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758343.254685:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758343.254687:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655106176:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758343.381956:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758343.830087:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758344.405959:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758344.853998:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758345.429992:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758345.878111:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758346.453999:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758346.902080:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758347.477971:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758347.926140:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758348.373986:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758348.373997:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758348.374001:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758348.374012:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11404 00000100:00080000:0.0:1787758348.374023:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758348.374031:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758348.374070:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655106304/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758348.374098:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758348.374101:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758348.374103:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758348.374106:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11404 00000100:00080000:0.0:1787758348.374109:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758348.374112:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758348.374120:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758348.374131:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655106432/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758348.374137:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758348.374140:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758348.374141:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758348.374144:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11404 00000100:00080000:0.0:1787758348.374146:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758348.374148:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758348.374151:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758348.374157:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588655106560/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758348.374164:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758348.374166:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758348.374167:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758348.374170:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11404 00000100:00080000:0.0:1787758348.374173:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758348.374176:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758348.374179:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758348.374186:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655106304:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787758348.374187:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588655106688/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758348.374200:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758348.374202:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758348.374203:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758348.374205:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11404 00000100:00080000:0.0:1787758348.374208:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758348.374211:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758348.374215:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758348.374223:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588655106816/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758348.374230:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758348.374232:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758348.374233:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758348.374235:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11404 00000100:00080000:0.0:1787758348.374245:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758348.374247:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758348.374250:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787758348.374255:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758348.374259:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655106944/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758348.374267:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758348.374270:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758348.374271:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758348.374274:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11404 00000100:00080000:0.0:1787758348.374276:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758348.374278:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758348.374281:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758348.374289:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655107072/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758348.374295:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758348.374298:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758348/real 1787758348] req@ffff8c05a9f74380 x1874588655106304/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758348.374299:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758348.374302:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758348.374314:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11404 00000100:00080000:0.0:1787758348.374318:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758348.374318:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655106304/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758348.374324:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758348.374327:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758348.374336:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655107200/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758348.374343:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758348.374349:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758348.374353:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655106304:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758348.374361:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655106432:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758348.374380:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758348.374389:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758348/real 1787758348] req@ffff8c05a9f76300 x1874588655106432/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758348.374396:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655106432/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758348.374402:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758348.374406:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758348.374409:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655106432:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758348.374413:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655106560:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758348.374427:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758348.374435:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758348/real 1787758348] req@ffff8c05a9f77b80 x1874588655106560/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758348.374446:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588655106560/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758348.374455:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758348.374460:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758348.374471:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655106560:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758348.374509:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655106688:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758348.374533:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758348.374566:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758348/real 1787758348] req@ffff8c05a9f76a00 x1874588655106688/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758348.374579:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588655106688/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758348.374586:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758348.374588:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758348.374591:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655106688:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758348.374596:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655106816:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758348.374614:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758348.374622:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758348/real 1787758348] req@ffff8c05a9f75c00 x1874588655106816/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758348.374629:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588655106816/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758348.374638:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758348.374640:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758348.374642:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655106816:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758348.374646:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655106944:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758348.374658:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758348.374666:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758348/real 1787758348] req@ffff8c05a9f77800 x1874588655106944/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758348.374674:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655106944/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758348.374687:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758348.374691:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758348.374694:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655106944:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758348.374698:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655107072:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758348.374713:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758348.374724:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758348/real 1787758348] req@ffff8c05a9f76680 x1874588655107072/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758348.374733:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655107072/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758348.374740:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758348.374744:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758348.374747:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655107072:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758348.374753:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655107200:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758348.374764:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758348.374774:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758348/real 1787758348] req@ffff8c05a9f74a80 x1874588655107200/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758348.374782:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655107200/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758348.374791:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758348.374795:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758348.374802:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655107200:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758348.502009:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758348.950065:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758349.525980:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758349.974181:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758350.549945:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758350.998021:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758351.573960:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758352.022086:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758352.597957:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758353.046078:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758353.493968:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758353.493979:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758353.493983:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758353.493993:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11409 00000100:00080000:3.0:1787758353.494004:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758353.494011:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758353.494047:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655107328/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758353.494084:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758353.494088:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758353.494090:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758353.494096:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11409 00000100:00080000:3.0:1787758353.494100:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758353.494104:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758353.494114:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758353.494132:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655107456/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758353.494145:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758353.494157:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655107328:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758353.494158:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758353.494164:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758353.494176:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11409 00000100:00080000:3.0:1787758353.494179:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758353.494184:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758353.494190:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758353.494206:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655107584/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758353.494217:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758353.494222:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758353.494225:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758353.494230:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11409 00000100:00080000:3.0:1787758353.494234:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758353.494239:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758353.494244:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758353.494257:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655107712/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787758353.494264:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758353.494280:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758353.494285:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758353.494287:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758353.494294:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11409 00000100:00080000:3.0:1787758353.494299:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758353.494303:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758353.494311:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758353.494320:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758353/real 1787758353] req@ffff8c05a9f79f80 x1874588655107328/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758353.494331:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655107840/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758353.494344:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655107328/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758353.494348:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758353.494361:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758353.494363:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758353.494365:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758353.494370:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787758353.494378:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11409 00000100:00100000:1.0:1787758353.494380:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655107328:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758353.494386:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758353.494393:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758353.494399:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758353.494409:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655107968/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758353.494418:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758353.494421:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758353.494424:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758353.494428:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11409 00000100:00100000:1.0:1787758353.494433:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655107456:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758353.494436:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758353.494444:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758353.494450:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758353.494481:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655108096/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758353.494493:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758353.494496:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758353.494498:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758353.494502:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11409 00000100:00080000:3.0:1787758353.494506:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787758353.494512:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758353.494516:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758353.494526:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758353.494531:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758353/real 1787758353] req@ffff8c05a9f7bb80 x1874588655107456/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758353.494543:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655108224/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758353.494545:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655107456/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758353.494613:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758353.494619:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758353.494625:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655107456:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758353.494636:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655107584:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758353.494655:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758353.494661:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758353/real 1787758353] req@ffff8c05a9f79500 x1874588655107584/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758353.494670:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655107584/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758353.494675:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758353.494677:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758353.494680:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655107584:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758353.494686:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655107712:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758353.494706:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758353.494718:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758353/real 1787758353] req@ffff8c05a9f7b800 x1874588655107712/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758353.494731:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655107712/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758353.494741:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758353.494743:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758353.494746:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655107712:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758353.494751:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655107840:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758353.494766:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758353.494774:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758353/real 1787758353] req@ffff8c05a9f78e00 x1874588655107840/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758353.494783:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655107840/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758353.494788:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758353.494790:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758353.494792:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655107840:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758353.494796:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655107968:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758353.494813:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758353.494820:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758353/real 1787758353] req@ffff8c05a9f7ad80 x1874588655107968/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758353.494828:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655107968/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758353.494834:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758353.494836:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758353.494838:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655107968:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758353.494853:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655108096:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758353.494865:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758353.494871:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758353/real 1787758353] req@ffff8c05a9f7b100 x1874588655108096/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758353.494920:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655108096/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758353.494931:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758353.494933:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758353.494935:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655108096:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758353.494940:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655108224:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758353.494952:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758353.494959:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758353/real 1787758353] req@ffff8c05a9f79180 x1874588655108224/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758353.494967:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655108224/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758353.494973:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758353.494974:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758353.494976:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655108224:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758353.621957:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758354.070074:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758354.645993:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758355.094039:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758355.670019:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758356.118142:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758356.693995:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758357.142090:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758357.717990:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758358.166234:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758358.614152:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758358.614161:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758358.614164:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758358.614173:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11415 00000100:00080000:0.0:1787758358.614184:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758358.614190:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758358.614225:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588655108352/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758358.614249:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758358.614252:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758358.614253:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758358.614256:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11415 00000100:00080000:0.0:1787758358.614258:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758358.614261:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758358.614268:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758358.614278:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655108480/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758358.614285:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758358.614287:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758358.614288:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758358.614290:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11415 00000100:00080000:0.0:1787758358.614292:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758358.614294:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758358.614296:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758358.614302:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588655108608/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758358.614307:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758358.614310:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758358.614311:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758358.614314:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11415 00000100:00080000:0.0:1787758358.614317:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758358.614319:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758358.614322:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758358.614328:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588655108736/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758358.614335:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758358.614338:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1787758358.614339:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655108352:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758358.614340:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758358.614348:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11415 00000100:00080000:0.0:1787758358.614350:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758358.614353:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758358.614356:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758358.614364:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588655108864/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758358.614383:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758358.614385:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758358.614387:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758358.614390:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11415 00000100:00080000:0.0:1787758358.614411:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758358.614415:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758358.614421:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758358.614434:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588655108992/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758358.614445:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758358.614448:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758358.614450:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758358.614455:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11415 00000100:00080000:0.0:1787758358.614460:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787758358.614464:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758358.614468:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758358.614476:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758358.614488:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588655109120/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758358.614500:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758358.614503:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758358.614505:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758358.614509:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11415 00000100:00080000:0.0:1787758358.614512:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758358.614516:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758358.614537:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758358.614543:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758358/real 1787758358] req@ffff8c05a9f75500 x1874588655108352/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758358.614548:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655109248/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758358.614575:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588655108352/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758358.614584:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758358.614589:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758358.614592:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655108352:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758358.614598:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655108480:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758358.614624:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758358.614633:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758358/real 1787758358] req@ffff8c05a9f75880 x1874588655108480/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758358.614641:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655108480/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758358.614647:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758358.614649:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758358.614651:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655108480:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758358.614655:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655108608:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758358.614665:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758358.614671:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758358/real 1787758358] req@ffff8c05a9f74000 x1874588655108608/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758358.614678:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588655108608/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758358.614683:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758358.614684:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758358.614686:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655108608:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758358.614706:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655108736:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758358.614718:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758358.614724:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758358/real 1787758358] req@ffff8c05a9f75180 x1874588655108736/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758358.614731:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588655108736/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758358.614736:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758358.614738:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758358.614740:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655108736:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758358.614743:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655108864:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758358.614752:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758358.614758:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758358/real 1787758358] req@ffff8c05a9f76d80 x1874588655108864/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758358.614765:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588655108864/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758358.614770:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758358.614771:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758358.614773:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655108864:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758358.614776:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655108992:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758358.614808:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758358.614819:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758358/real 1787758358] req@ffff8c05a9f77100 x1874588655108992/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758358.614842:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588655108992/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758358.614852:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758358.614855:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758358.614859:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655108992:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758358.614865:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655109120:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758358.614883:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758358.614951:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758358/real 1787758358] req@ffff8c05a9f74e00 x1874588655109120/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758358.614965:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588655109120/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758358.614974:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758358.614977:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758358.614980:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655109120:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758358.614986:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655109248:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758358.615006:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758358.615016:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758358/real 1787758358] req@ffff8c05a9f74a80 x1874588655109248/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758358.615027:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655109248/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758358.615041:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758358.615044:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758358.615048:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655109248:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758358.742045:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758359.190183:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758359.766001:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758360.214313:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758360.790108:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758361.238170:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758361.814049:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758362.262220:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758362.838071:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758363.286172:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758363.734095:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758363.734109:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758363.734116:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758363.734130:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11420 00000100:00080000:3.0:1787758363.734146:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758363.734157:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758363.734213:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655109376/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758363.734267:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758363.734276:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758363.734304:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758363.734317:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11420 00000100:00080000:3.0:1787758363.734326:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758363.734336:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758363.734350:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758363.734373:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655109504/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758363.734389:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655109376:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758363.734396:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758363.734409:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758363.734412:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758363.734418:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11420 00000100:00080000:3.0:1787758363.734423:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758363.734427:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758363.734434:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758363.734450:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655109632/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758363.734464:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758363.734469:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758363.734472:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758363.734476:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11420 00000100:00080000:3.0:1787758363.734482:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758363.734486:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758363.734491:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787758363.734503:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758363.734510:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655109760/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758363.734526:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758363.734531:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758363.734533:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758363.734538:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11420 00000100:00080000:3.0:1787758363.734542:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758363.734547:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758363.734553:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758363.734566:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655109888/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758363.734577:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758363.734582:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787758363.734589:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758363/real 1787758363] req@ffff8c05a9f79180 x1874588655109376/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758363.734590:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758363.734604:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11420 00000100:00080000:3.0:1787758363.734614:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758363.734626:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758363.734635:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758363.734643:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655109376/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758363.734659:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655110016/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758363.734666:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758363.734682:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758363.734688:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758363.734691:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758363.734697:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787758363.734698:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11420 00000100:00080000:3.0:1787758363.734710:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758363.734716:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655109376:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758363.734723:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758363.734735:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758363.734749:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655110144/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758363.734764:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758363.734766:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655109504:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758363.734776:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758363.734782:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758363.734789:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11420 00000100:00080000:3.0:1787758363.734794:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758363.734799:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758363.734806:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787758363.734814:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758363.734828:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655110272/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758363.734842:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758363/real 1787758363] req@ffff8c05a9f7b100 x1874588655109504/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758363.734863:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655109504/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758363.734943:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758363.734948:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758363.734955:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655109504:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758363.734965:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655109632:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758363.734994:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758363.735009:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758363/real 1787758363] req@ffff8c05a9f7ad80 x1874588655109632/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758363.735024:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655109632/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758363.735034:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758363.735038:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758363.735042:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655109632:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758363.735050:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655109760:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758363.735084:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758363.735105:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758363/real 1787758363] req@ffff8c05a9f78e00 x1874588655109760/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758363.735130:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655109760/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758363.735148:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758363.735154:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758363.735162:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655109760:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758363.735172:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655109888:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758363.735201:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758363.735216:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758363/real 1787758363] req@ffff8c05a9f7b800 x1874588655109888/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758363.735231:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655109888/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758363.735241:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758363.735244:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758363.735248:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655109888:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758363.735255:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655110016:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758363.735273:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758363.735315:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758363/real 1787758363] req@ffff8c05a9f79500 x1874588655110016/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758363.735336:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655110016/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758363.735352:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758363.735359:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758363.735365:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655110016:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758363.735404:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655110144:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758363.735444:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758363.735461:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758363/real 1787758363] req@ffff8c05a9f7bb80 x1874588655110144/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758363.735477:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655110144/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758363.735487:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758363.735491:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758363.735495:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655110144:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758363.735502:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655110272:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758363.735522:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758363.735534:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758363/real 1787758363] req@ffff8c05a9f79f80 x1874588655110272/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758363.735548:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655110272/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758363.735557:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758363.735561:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758363.735564:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655110272:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758363.862052:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758364.310245:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758364.886093:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758365.334204:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758365.910097:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758366.358320:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758366.934027:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758367.382160:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758367.958044:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758368.406296:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758368.854093:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758368.854109:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787758368.854116:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758368.854132:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11425 00000100:00080000:2.0:1787758368.854150:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758368.854164:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758368.854267:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588655110400/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758368.854318:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758368.854324:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758368.854327:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758368.854333:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11425 00000100:00080000:2.0:1787758368.854339:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758368.854347:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758368.854365:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758368.854388:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588655110528/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758368.854406:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758368.854412:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655110400:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787758368.854413:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758368.854421:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758368.854443:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11425 00000100:00080000:2.0:1787758368.854448:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758368.854454:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758368.854463:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758368.854490:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588655110656/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758368.854504:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758368.854510:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758368.854514:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758368.854520:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11425 00000100:00080000:2.0:1787758368.854526:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758368.854533:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758368.854540:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787758368.854550:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758368.854561:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588655110784/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758368.854579:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758368.854586:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758368.854589:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758368.854597:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11425 00000100:00080000:2.0:1787758368.854608:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758368.854618:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758368.854626:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758368/real 1787758368] req@ffff8c05b2e70e00 x1874588655110400/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787758368.854633:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758368.854694:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588655110912/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758368.854695:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588655110400/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758368.854803:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758368.854807:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758368.854810:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758368.854818:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758368.854830:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11425 00000100:00080000:1.0:1787758368.854831:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787758368.854842:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758368.854852:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655110400:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787758368.854853:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758368.854940:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758368.854971:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655110528:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758368.854975:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588655111040/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758368.855021:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758368.855029:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787758368.855035:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787758368.855048:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787758368.855050:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11425 00000100:00080000:2.0:1787758368.855069:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758368.855084:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758368.855085:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758368/real 1787758368] req@ffff8c05b2e70700 x1874588655110528/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787758368.855104:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758368.855116:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588655110528/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758368.855132:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588655111168/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758368.855133:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758368.855161:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787758368.855220:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758368.855226:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655110528:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787758368.855235:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758368.855243:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758368.855245:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655110656:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787758368.855267:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11425 00000100:00080000:2.0:1787758368.855279:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758368.855289:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:1.0:1787758368.855302:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:2.0:1787758368.855310:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758368.855325:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758368/real 1787758368] req@ffff8c05b2e73480 x1874588655110656/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758368.855341:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588655111296/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758368.855354:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588655110656/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758368.855397:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758368.855402:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758368.855407:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655110656:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758368.855417:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655110784:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758368.855439:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758368.855450:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758368/real 1787758368] req@ffff8c05b2e71500 x1874588655110784/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758368.855463:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588655110784/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758368.855472:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758368.855476:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758368.855479:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655110784:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758368.855486:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655110912:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758368.855530:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758368.855541:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758368/real 1787758368] req@ffff8c05b2e73b80 x1874588655110912/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758368.855555:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588655110912/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758368.855565:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758368.855568:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758368.855573:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655110912:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758368.855598:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655111040:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758368.855618:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758368.855630:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758368/real 1787758368] req@ffff8c05b2e71c00 x1874588655111040/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758368.855644:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588655111040/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758368.855654:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758368.855657:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758368.855661:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655111040:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758368.855668:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655111168:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758368.855687:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758368.855698:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758368/real 1787758368] req@ffff8c05b2e71880 x1874588655111168/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758368.855712:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588655111168/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758368.855722:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758368.855725:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758368.855728:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655111168:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758368.855735:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655111296:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758368.855752:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758368.855764:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758368/real 1787758368] req@ffff8c05b2e70380 x1874588655111296/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758368.855778:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588655111296/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758368.855788:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758368.855792:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758368.855795:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655111296:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758368.982039:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758369.430197:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758370.006044:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758370.454314:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758371.030090:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758371.478224:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758372.054090:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758372.502324:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758373.078041:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758373.526172:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758373.974067:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758373.974081:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758373.974087:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758373.974122:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11430 00000100:00080000:0.0:1787758373.974139:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758373.974150:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758373.974197:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655111424/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758373.974245:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758373.974250:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758373.974253:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758373.974258:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11430 00000100:00080000:0.0:1787758373.974263:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758373.974269:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758373.974281:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758373.974300:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655111552/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758373.974314:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758373.974318:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758373.974321:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758373.974326:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11430 00000100:00080000:0.0:1787758373.974330:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758373.974334:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758373.974344:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655111424:192.168.201.107@tcp:250:kworker.0 00000008:00100000:0.0:1787758373.974346:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758373.974369:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588655111680/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758373.974384:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758373.974388:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758373.974391:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758373.974399:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11430 00000100:00080000:0.0:1787758373.974406:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758373.974414:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758373.974423:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787758373.974451:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758373.974454:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588655111808/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758373.974480:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758373.974488:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758373.974491:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758373.974500:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11430 00000100:00080000:0.0:1787758373.974508:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758373.974515:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758373.974523:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758373.974533:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758373/real 1787758373] req@ffff8c05a9f76680 x1874588655111424/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758373.974549:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588655111936/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758373.974584:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758373.974588:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758373.974591:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758373.974598:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655111424/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758373.974604:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11430 00000100:00080000:0.0:1787758373.974616:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758373.974629:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:1.0:1787758373.974632:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787758373.974636:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787758373.974644:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758373.974653:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655111424:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787758373.974660:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655112064/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758373.974672:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655111552:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758373.974680:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758373.974686:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758373.974689:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758373.974696:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11430 00000100:00080000:0.0:1787758373.974701:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758373.974706:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:1.0:1787758373.974723:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:0.0:1787758373.974729:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758373.974745:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758373/real 1787758373] req@ffff8c05a9f77800 x1874588655111552/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758373.974763:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655112192/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758373.974792:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655111552/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758373.974806:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758373.974811:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758373.974819:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758373.974832:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758373.974835:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787758373.974851:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11430 00000100:00100000:1.0:1787758373.974852:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655111552:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758373.974867:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758373.974951:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655111680:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758373.974965:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758373.974976:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787758373.974987:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758373.975004:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655112320/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758373.975011:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758373/real 1787758373] req@ffff8c05a9f75c00 x1874588655111680/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758373.975062:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588655111680/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758373.975080:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758373.975088:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758373.975097:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655111680:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758373.975138:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655111808:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758373.975181:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758373.975205:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758373/real 1787758373] req@ffff8c05a9f76a00 x1874588655111808/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758373.975224:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588655111808/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758373.975243:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758373.975247:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758373.975251:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655111808:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758373.975258:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655111936:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758373.975286:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758373.975299:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758373/real 1787758373] req@ffff8c05a9f77b80 x1874588655111936/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758373.975313:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588655111936/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758373.975323:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758373.975327:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758373.975330:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655111936:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758373.975337:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655112064:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758373.975356:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758373.975367:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758373/real 1787758373] req@ffff8c05a9f76300 x1874588655112064/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758373.975381:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655112064/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758373.975390:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758373.975394:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758373.975398:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655112064:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758373.975405:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655112192:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758373.975424:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758373.975435:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758373/real 1787758373] req@ffff8c05a9f74380 x1874588655112192/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758373.975448:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655112192/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758373.975464:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758373.975470:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758373.975476:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655112192:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758373.975512:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655112320:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758373.975554:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758373.975576:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758373/real 1787758373] req@ffff8c05a9f77800 x1874588655112320/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758373.975592:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655112320/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758373.975602:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758373.975606:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758373.975610:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655112320:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758374.102045:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758374.550229:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758375.125997:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758375.574110:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758376.150044:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758376.598227:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758377.173999:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758377.622143:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758378.198083:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758378.646275:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758379.094127:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758379.094139:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758379.094144:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758379.094156:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11435 00000100:00080000:1.0:1787758379.094170:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758379.094179:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758379.094228:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588655112448/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758379.094274:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758379.094278:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758379.094281:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758379.094286:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11435 00000100:00080000:1.0:1787758379.094290:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758379.094295:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758379.094307:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758379.094322:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655112576/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758379.094335:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758379.094340:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758379.094342:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758379.094346:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11435 00000100:00080000:1.0:1787758379.094351:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758379.094355:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758379.094364:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758379.094379:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655112704/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758379.094392:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758379.094396:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758379.094398:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758379.094405:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11435 00000100:00080000:1.0:1787758379.094411:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758379.094417:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758379.094424:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758379.094439:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655112448:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758379.094454:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655112832/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758379.094471:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758379.094477:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758379.094479:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758379.094486:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11435 00000100:00080000:1.0:1787758379.094490:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758379.094496:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758379.094503:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758379.094530:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655112960/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758379.094541:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758379.094545:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758379.094548:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758379.094554:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11435 00000100:00080000:1.0:1787758379.094562:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758379.094566:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758379.094573:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787758379.094581:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758379.094589:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655113088/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758379.094603:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758379.094608:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758379.094610:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758379.094616:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11435 00000100:00080000:1.0:1787758379.094620:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758379.094624:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758379.094631:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758379.094647:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655113216/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758379.094658:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758379.094662:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758379.094665:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758379.094673:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11435 00000100:00100000:3.0:1787758379.094681:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758379/real 1787758379] req@ffff8c05b2d9a300 x1874588655112448/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758379.094682:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758379.094705:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758379.094731:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758379.094755:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655113344/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787758379.094765:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588655112448/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758379.094813:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758379.094820:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758379.094827:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655112448:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787758379.094840:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655112576:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758379.094874:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758379.094939:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758379/real 1787758379] req@ffff8c05b2d98380 x1874588655112576/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758379.094956:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655112576/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758379.094968:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758379.094972:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758379.094977:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655112576:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758379.094985:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655112704:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758379.095038:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758379.095055:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758379/real 1787758379] req@ffff8c05b2d9aa00 x1874588655112704/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758379.095070:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655112704/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758379.095081:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758379.095085:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758379.095089:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655112704:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758379.095128:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655112832:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758379.095153:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758379.095165:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758379/real 1787758379] req@ffff8c05b2d9bb80 x1874588655112832/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758379.095189:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655112832/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758379.095200:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758379.095203:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758379.095207:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655112832:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758379.095214:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655112960:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758379.095235:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758379.095247:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758379/real 1787758379] req@ffff8c05b2d98700 x1874588655112960/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758379.095261:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655112960/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758379.095271:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758379.095275:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758379.095278:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655112960:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758379.095285:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655113088:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758379.095304:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758379.095315:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758379/real 1787758379] req@ffff8c05b2d9b100 x1874588655113088/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758379.095329:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655113088/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758379.095348:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758379.095352:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758379.095356:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655113088:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758379.095363:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655113216:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758379.095381:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758379.095392:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758379/real 1787758379] req@ffff8c05b2d99880 x1874588655113216/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758379.095406:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655113216/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758379.095417:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758379.095420:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758379.095424:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655113216:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758379.095430:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655113344:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758379.095450:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758379.095461:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758379/real 1787758379] req@ffff8c05b2d99f80 x1874588655113344/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758379.095475:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655113344/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758379.095485:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758379.095489:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758379.095494:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655113344:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758379.222042:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758379.670029:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758380.245965:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758380.694281:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758381.270000:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758381.718112:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758382.294023:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758382.742274:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758383.318020:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758383.766088:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758384.214022:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758384.214031:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758384.214035:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758384.214044:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11440 00000100:00080000:3.0:1787758384.214054:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758384.214060:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758384.214095:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655113472/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758384.214127:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758384.214131:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758384.214132:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758384.214139:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11440 00000100:00080000:3.0:1787758384.214142:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758384.214145:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758384.214155:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758384.214166:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655113600/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758384.214178:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758384.214181:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758384.214182:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758384.214184:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11440 00000100:00080000:3.0:1787758384.214186:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758384.214190:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758384.214195:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758384.214210:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655113728/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758384.214224:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655113472:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758384.214224:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758384.214240:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758384.214246:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758384.214250:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11440 00000100:00080000:3.0:1787758384.214252:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758384.214255:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758384.214258:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758384.214267:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655113856/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758384.214275:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758384.214278:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758384.214279:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758384.214283:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11440 00000100:00080000:3.0:1787758384.214290:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758384.214294:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758384.214298:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758384.214307:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655113984/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758384.214314:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758384.214317:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758384.214320:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758384.214323:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11440 00000100:00080000:3.0:1787758384.214325:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758384.214328:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758384.214330:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787758384.214337:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758384.214338:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655114112/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758384.214350:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758384.214352:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758384.214354:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758384.214357:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11440 00000100:00080000:3.0:1787758384.214360:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758384.214363:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758384.214366:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758384.214375:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655114240/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758384.214386:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758384.214388:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758384.214390:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758384.214393:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11440 00000100:00080000:3.0:1787758384.214397:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787758384.214409:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758384/real 1787758384] req@ffff8c05a9f7b480 x1874588655113472/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758384.214431:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655113472/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758384.214445:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758384.214451:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758384.214457:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655113472:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787758384.214472:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655113600:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758384.214475:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758384.214488:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787758384.214515:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758384.214533:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758384/real 1787758384] req@ffff8c05a9f78380 x1874588655113600/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758384.214546:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655113600/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758384.214554:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758384.214558:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758384.214562:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655113600:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758384.214598:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655113728:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758384.214614:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655114368/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787758384.214625:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758384.214648:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758384/real 1787758384] req@ffff8c05a9f78a80 x1874588655113728/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758384.214667:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655113728/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758384.214677:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758384.214682:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758384.214686:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655113728:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758384.214694:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655113856:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758384.214724:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758384.214750:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758384/real 1787758384] req@ffff8c05a9f7a680 x1874588655113856/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758384.214770:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655113856/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758384.214779:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758384.214783:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758384.214788:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655113856:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758384.214795:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655113984:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758384.214816:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758384.214827:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758384/real 1787758384] req@ffff8c05a9f7aa00 x1874588655113984/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758384.214839:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655113984/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758384.214848:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758384.214851:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758384.214855:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655113984:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758384.214862:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655114112:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758384.214938:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758384.214964:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758384/real 1787758384] req@ffff8c05a9f7a300 x1874588655114112/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758384.214978:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655114112/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758384.214993:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758384.214998:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758384.215001:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655114112:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758384.215009:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655114240:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758384.215030:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758384.215041:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758384/real 1787758384] req@ffff8c05a9f79880 x1874588655114240/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758384.215052:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655114240/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758384.215061:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758384.215064:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758384.215067:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655114240:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758384.215089:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655114368:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758384.215107:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758384.215117:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758384/real 1787758384] req@ffff8c05a9f78700 x1874588655114368/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758384.215128:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655114368/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758384.215136:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758384.215139:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758384.215143:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655114368:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758384.341957:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758384.790150:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758385.366074:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758385.814065:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758386.390047:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758386.838192:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758387.413973:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758387.862083:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758388.438063:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758388.886057:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758389.333962:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758389.333973:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758389.333977:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758389.333986:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11445 00000100:00080000:3.0:1787758389.333996:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758389.334002:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758389.334040:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655114496/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758389.334072:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758389.334076:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758389.334077:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758389.334080:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11445 00000100:00080000:3.0:1787758389.334083:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758389.334086:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758389.334095:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758389.334106:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655114624/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758389.334113:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758389.334116:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758389.334117:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758389.334120:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11445 00000100:00080000:3.0:1787758389.334121:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758389.334123:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758389.334126:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758389.334137:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655114752/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758389.334145:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758389.334147:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758389.334150:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758389.334156:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11445 00000100:00080000:3.0:1787758389.334158:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758389.334163:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758389.334168:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758389.334178:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655114496:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787758389.334179:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655114880/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758389.334194:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758389.334196:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758389.334198:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758389.334201:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11445 00000100:00080000:3.0:1787758389.334203:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758389.334206:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758389.334208:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758389.334217:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655115008/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758389.334223:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758389.334225:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758389.334226:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758389.334229:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11445 00000100:00080000:3.0:1787758389.334231:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758389.334233:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758389.334237:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758389.334244:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655115136/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787758389.334251:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758389.334253:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758389.334257:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758389.334258:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758389.334262:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11445 00000100:00080000:3.0:1787758389.334266:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758389.334268:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758389.334271:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758389.334279:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655115264/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758389.334284:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758389.334286:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758389.334288:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758389.334291:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11445 00000100:00100000:1.0:1787758389.334295:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758389/real 1787758389] req@ffff8c05a9f78700 x1874588655114496/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758389.334296:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758389.334303:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758389.334310:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758389.334320:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655114496/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758389.334322:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655115392/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758389.334351:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758389.334356:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758389.334359:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655114496:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758389.334367:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655114624:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758389.334384:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758389.334394:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758389/real 1787758389] req@ffff8c05a9f79880 x1874588655114624/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758389.334402:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655114624/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758389.334407:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758389.334409:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758389.334412:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655114624:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758389.334416:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655114752:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758389.334428:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758389.334433:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758389/real 1787758389] req@ffff8c05a9f7a300 x1874588655114752/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758389.334440:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655114752/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758389.334445:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758389.334447:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758389.334449:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655114752:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758389.334468:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655114880:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758389.334479:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758389.334485:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758389/real 1787758389] req@ffff8c05a9f7aa00 x1874588655114880/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758389.334492:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655114880/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758389.334497:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758389.334499:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758389.334500:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655114880:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758389.334504:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655115008:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758389.334514:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758389.334520:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758389/real 1787758389] req@ffff8c05a9f7a680 x1874588655115008/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758389.334527:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655115008/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758389.334532:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758389.334534:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758389.334535:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655115008:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758389.334538:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655115136:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758389.334547:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758389.334553:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758389/real 1787758389] req@ffff8c05a9f78a80 x1874588655115136/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758389.334559:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655115136/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758389.334564:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758389.334566:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758389.334568:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655115136:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758389.334571:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655115264:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758389.334580:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758389.334585:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758389/real 1787758389] req@ffff8c05a9f78380 x1874588655115264/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758389.334592:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655115264/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758389.334608:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758389.334610:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758389.334612:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655115264:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758389.334616:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655115392:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758389.334628:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758389.334634:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758389/real 1787758389] req@ffff8c05a9f7b480 x1874588655115392/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758389.334642:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655115392/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758389.334647:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758389.334649:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758389.334651:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655115392:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758389.461981:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758389.910125:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758390.485999:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758390.934046:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758391.509964:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758391.957984:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758392.533951:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758392.982069:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758393.557999:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758394.005990:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758394.453974:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758394.453982:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787758394.453986:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758394.453994:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11450 00000100:00080000:2.0:1787758394.454003:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758394.454010:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758394.454050:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73800 x1874588655115520/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758394.454078:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758394.454081:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758394.454082:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758394.454085:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11450 00000100:00080000:2.0:1787758394.454089:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758394.454093:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758394.454101:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758394.454116:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588655115648/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758394.454117:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655115520:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787758394.454132:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758394.454135:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758394.454136:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758394.454139:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11450 00000100:00080000:2.0:1787758394.454141:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758394.454143:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758394.454147:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758394.454156:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588655115776/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758394.454163:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758394.454165:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758394.454166:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758394.454168:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11450 00000100:00080000:2.0:1787758394.454171:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758394.454174:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758394.454176:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758394.454191:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588655115904/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758394.454201:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758394.454203:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758394.454204:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758394.454206:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11450 00000100:00080000:2.0:1787758394.454208:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758394.454210:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758394.454213:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787758394.454218:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758394.454223:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588655116032/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758394.454230:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758394.454232:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758394.454234:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758394.454237:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11450 00000100:00080000:2.0:1787758394.454239:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758394.454241:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758394.454244:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758394.454251:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588655116160/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758394.454256:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758394.454258:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787758394.454259:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758394.454262:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11450 00000100:00080000:2.0:1787758394.454265:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758394.454269:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758394/real 1787758394] req@ffff8c05b2e73800 x1874588655115520/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758394.454272:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758394.454279:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758394.454291:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588655116288/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758394.454297:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73800 x1874588655115520/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758394.454301:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758394.454308:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758394.454311:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758394.454318:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11450 00000100:00080000:2.0:1787758394.454324:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758394.454330:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:1.0:1787758394.454332:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787758394.454334:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787758394.454339:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758394.454345:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588655116416/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758394.454346:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655115520:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758394.454380:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655115648:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758394.454407:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758394.454418:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758394/real 1787758394] req@ffff8c05b2e70380 x1874588655115648/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758394.454427:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588655115648/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758394.454433:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758394.454436:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758394.454438:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655115648:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758394.454442:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655115776:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758394.454464:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758394.454471:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758394/real 1787758394] req@ffff8c05b2e71880 x1874588655115776/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758394.454478:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588655115776/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758394.454484:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758394.454486:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758394.454489:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655115776:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758394.454493:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655115904:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758394.454504:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758394.454510:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758394/real 1787758394] req@ffff8c05b2e71c00 x1874588655115904/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758394.454517:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588655115904/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758394.454523:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758394.454525:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758394.454526:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655115904:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758394.454530:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655116032:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758394.454540:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758394.454546:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758394/real 1787758394] req@ffff8c05b2e73b80 x1874588655116032/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758394.454553:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588655116032/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758394.454558:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758394.454559:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758394.454561:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655116032:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758394.454564:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655116160:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758394.454573:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758394.454578:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758394/real 1787758394] req@ffff8c05b2e71500 x1874588655116160/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758394.454586:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588655116160/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758394.454591:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758394.454592:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758394.454594:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655116160:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758394.454597:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655116288:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758394.454606:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758394.454611:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758394/real 1787758394] req@ffff8c05b2e73480 x1874588655116288/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758394.454618:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588655116288/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758394.454623:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758394.454625:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758394.454626:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655116288:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758394.454631:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655116416:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758394.454647:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758394.454658:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758394/real 1787758394] req@ffff8c05b2e70700 x1874588655116416/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758394.454670:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588655116416/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758394.454685:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758394.454688:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758394.454690:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655116416:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758394.581951:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758395.030064:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758395.605958:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758396.054124:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758396.629980:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758397.078173:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758397.654001:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758398.102037:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758398.677945:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758399.126066:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758399.574049:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758399.574059:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758399.574064:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758399.574074:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11455 00000100:00080000:3.0:1787758399.574086:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758399.574093:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758399.574132:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655116544/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758399.574162:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758399.574166:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758399.574168:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758399.574172:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11455 00000100:00080000:3.0:1787758399.574175:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758399.574178:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758399.574188:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758399.574199:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655116672/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758399.574209:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758399.574212:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758399.574214:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758399.574217:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11455 00000100:00080000:3.0:1787758399.574220:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758399.574226:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758399.574231:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758399.574240:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655116800/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758399.574250:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758399.574257:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787758399.574259:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655116544:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758399.574260:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758399.574271:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11455 00000100:00080000:3.0:1787758399.574274:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758399.574278:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758399.574283:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758399.574297:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655116928/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758399.574307:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758399.574310:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758399.574312:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758399.574316:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11455 00000100:00080000:3.0:1787758399.574320:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758399.574323:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758399.574327:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758399.574340:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655117056/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758399.574348:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758399.574352:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758399.574354:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758399.574357:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11455 00000100:00080000:3.0:1787758399.574361:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758399.574364:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758399.574368:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758399.574378:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655117184/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758399.574386:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758399.574389:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758399.574391:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758399.574399:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655116672:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758399.574399:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11455 00000100:00080000:3.0:1787758399.574407:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758399.574417:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758399.574427:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758399.574449:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655117312/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758399.574464:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758399.574469:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787758399.574473:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655116800:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758399.574476:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758399.574489:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11455 00000100:00080000:3.0:1787758399.574497:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758399.574504:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758399.574512:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655116928:192.168.201.107@tcp:8:kworker.0 00000008:00100000:3.0:1787758399.574514:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758399.574539:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655117440/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758399.574543:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655117056:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758399.574580:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655117184:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758399.574604:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655117312:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758399.574638:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655117440:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758399.702067:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758400.150071:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758400.725963:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758401.174063:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758401.749953:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758402.198064:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787758402.646120:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.107:988 00000400:00000100:0.0:1787758402.646178:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.107@tcp at host 192.168.201.107 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787758402.646210:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787758402.646261:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787758402.646272:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787758402.646280:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787758402.646290:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787758402.646315:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787758402.646327:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:1.0:1787758402.646328:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758399/real 1787758402] req@ffff8c05a9f79500 x1874588655116800/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787758402.646340:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:1.0:1787758402.646370:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655116800/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758402.646389:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758402.646395:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758402.646400:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655116800:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758402.646415:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758399/real 1787758402] req@ffff8c05a9f7b800 x1874588655116928/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758402.646430:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655116928/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758402.646439:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758402.646443:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758402.646447:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655116928:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758402.646455:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758399/real 1787758402] req@ffff8c05a9f78e00 x1874588655117056/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758402.646467:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655117056/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758402.646476:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758402.646479:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758402.646482:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655117056:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758402.646489:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758399/real 1787758402] req@ffff8c05a9f7ad80 x1874588655117184/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758402.646501:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655117184/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758402.646508:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758402.646537:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758402.646542:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655117184:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758402.646549:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758399/real 1787758402] req@ffff8c05a9f7b100 x1874588655117312/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758402.646562:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655117312/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758402.646572:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758402.646590:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758402.646594:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655117312:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758402.646616:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758399/real 1787758402] req@ffff8c05a9f79f80 x1874588655116544/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758402.646631:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655116544/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758402.646639:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758402.646643:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758402.646647:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655116544:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758402.646655:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758399/real 1787758402] req@ffff8c05a9f7bb80 x1874588655116672/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758402.646668:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655116672/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758402.646676:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758402.646680:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758402.646684:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655116672:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758402.646714:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758399/real 1787758402] req@ffff8c05a9f79180 x1874588655117440/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758402.646728:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655117440/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758402.646737:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758402.646741:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758402.646745:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655117440:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758402.773950:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758403.222047:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758403.797975:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758404.246066:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758404.694022:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758404.694037:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758404.694042:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758404.694054:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11461 00000100:00080000:3.0:1787758404.694070:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758404.694079:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758404.694124:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655117568/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758404.694161:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758404.694165:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758404.694167:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758404.694172:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11461 00000100:00080000:3.0:1787758404.694175:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758404.694181:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758404.694192:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758404.694208:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655117696/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758404.694223:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758404.694226:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758404.694229:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758404.694237:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11461 00000100:00100000:1.0:1787758404.694240:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655117568:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758404.694243:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758404.694256:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758404.694262:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758404.694277:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655117824/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758404.694288:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758404.694293:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758404.694295:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758404.694301:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11461 00000100:00080000:3.0:1787758404.694304:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758404.694309:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758404.694315:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758404.694327:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655117952/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787758404.694350:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758404.694354:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758404.694360:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758404.694362:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758404.694371:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11461 00000100:00080000:3.0:1787758404.694377:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758404.694383:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758404.694391:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758404.694410:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655118080/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758404.694426:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758404/real 1787758404] req@ffff8c05a9f79180 x1874588655117568/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758404.694428:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758404.694446:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758404.694451:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758404.694470:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11461 00000100:00080000:3.0:1787758404.694481:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758404.694489:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758404.694494:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758404.694497:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655117568/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758404.694515:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655118208/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758404.694528:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758404.694531:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758404.694534:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758404.694539:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758404.694550:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787758404.694550:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11461 00000100:00080000:3.0:1787758404.694577:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758404.694578:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655117568:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758404.694592:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758404.694601:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758404.694616:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655118336/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758404.694628:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655117696:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758404.694663:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758404.694668:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758404.694670:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758404.694678:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11461 00000100:00080000:3.0:1787758404.694684:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758404.694690:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:1.0:1787758404.694699:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:3.0:1787758404.694699:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758404.694726:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655118464/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758404.694735:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758404/real 1787758404] req@ffff8c05a9f7bb80 x1874588655117696/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758404.694769:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655117696/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758404.694779:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758404.694786:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758404.694794:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655117696:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758404.694805:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655117824:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758404.694830:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758404.694842:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758404/real 1787758404] req@ffff8c05a9f79f80 x1874588655117824/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758404.694864:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655117824/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758404.694941:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758404.694948:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758404.694952:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655117824:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758404.694960:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655117952:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758404.694986:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758404.695001:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758404/real 1787758404] req@ffff8c05a9f7b100 x1874588655117952/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758404.695020:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655117952/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758404.695029:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758404.695032:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758404.695035:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655117952:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758404.695043:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655118080:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758404.695065:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758404.695078:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758404/real 1787758404] req@ffff8c05a9f7ad80 x1874588655118080/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758404.695089:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655118080/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758404.695094:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758404.695097:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758404.695100:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655118080:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758404.695106:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655118208:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758404.695135:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758404.695147:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758404/real 1787758404] req@ffff8c05a9f78e00 x1874588655118208/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758404.695160:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655118208/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758404.695174:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758404.695178:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758404.695181:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655118208:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758404.695206:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655118336:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758404.695228:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758404.695241:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758404/real 1787758404] req@ffff8c05a9f7b800 x1874588655118336/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758404.695253:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655118336/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758404.695262:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758404.695266:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758404.695270:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655118336:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758404.695277:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655118464:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758404.695300:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758404.695314:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758404/real 1787758404] req@ffff8c05a9f79500 x1874588655118464/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758404.695327:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655118464/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758404.695337:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758404.695339:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758404.695342:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655118464:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758404.822008:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758405.270086:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758405.845997:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758406.294040:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758406.869982:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758407.318100:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758407.893981:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758408.342058:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758408.918000:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758409.366114:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758409.813966:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758409.813977:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758409.813981:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758409.813992:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11466 00000100:00080000:3.0:1787758409.814003:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758409.814011:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758409.814044:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655118592/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758409.814068:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758409.814071:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758409.814072:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758409.814076:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11466 00000100:00080000:3.0:1787758409.814078:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758409.814081:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758409.814088:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758409.814098:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655118720/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758409.814107:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758409.814109:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758409.814110:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758409.814114:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11466 00000100:00080000:3.0:1787758409.814118:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758409.814119:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655118592:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758409.814121:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758409.814127:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758409.814134:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655118848/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758409.814141:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758409.814143:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758409.814145:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758409.814147:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11466 00000100:00080000:3.0:1787758409.814149:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758409.814152:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758409.814156:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758409.814165:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655118976/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758409.814171:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758409.814181:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758409.814184:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787758409.814186:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758409.814189:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11466 00000100:00080000:3.0:1787758409.814194:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758409.814197:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758409.814202:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758409.814211:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655119104/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758409.814218:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758409.814220:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758409.814221:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758409.814224:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11466 00000100:00080000:3.0:1787758409.814227:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758409.814230:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758409/real 1787758409] req@ffff8c05a9f7b480 x1874588655118592/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758409.814232:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758409.814238:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758409.814246:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655119232/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758409.814265:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758409.814268:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758409.814270:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758409.814273:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655118592/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758409.814276:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11466 00000100:00080000:3.0:1787758409.814281:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758409.814287:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758409.814292:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787758409.814297:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787758409.814301:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655119360/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758409.814306:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787758409.814314:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758409.814317:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655118592:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758409.814318:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758409.814321:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758409.814328:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11466 00000100:00080000:3.0:1787758409.814331:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758409.814333:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758409.814336:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758409.814342:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655119488/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758409.814352:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655118720:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758409.814375:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758409.814388:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758409/real 1787758409] req@ffff8c05a9f78380 x1874588655118720/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758409.814435:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655118720/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758409.814448:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758409.814453:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758409.814457:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655118720:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758409.814466:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655118848:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758409.814524:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758409.814538:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758409/real 1787758409] req@ffff8c05a9f78a80 x1874588655118848/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758409.814552:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655118848/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758409.814561:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758409.814565:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758409.814569:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655118848:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758409.814576:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655118976:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758409.814599:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758409.814610:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758409/real 1787758409] req@ffff8c05a9f7a680 x1874588655118976/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758409.814623:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655118976/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758409.814633:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758409.814636:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758409.814640:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655118976:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758409.814647:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655119104:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758409.814668:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758409.814681:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758409/real 1787758409] req@ffff8c05a9f7aa00 x1874588655119104/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758409.814702:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655119104/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758409.814712:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758409.814715:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758409.814719:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655119104:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758409.814725:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655119232:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758409.814742:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758409.814750:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758409/real 1787758409] req@ffff8c05a9f7a300 x1874588655119232/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758409.814758:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655119232/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758409.814763:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758409.814765:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758409.814767:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655119232:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758409.814771:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655119360:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758409.814781:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758409.814787:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758409/real 1787758409] req@ffff8c05a9f79880 x1874588655119360/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758409.814794:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655119360/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758409.814804:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758409.814806:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758409.814808:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655119360:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758409.814822:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655119488:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758409.814832:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758409.814837:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758409/real 1787758409] req@ffff8c05a9f78700 x1874588655119488/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758409.814844:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655119488/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758409.814850:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758409.814851:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758409.814853:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655119488:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758409.941978:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758410.390049:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758410.965952:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758411.414067:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758411.989956:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758412.438005:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758413.013992:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758413.462081:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758414.037955:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758414.486016:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758414.933941:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758414.933949:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758414.933953:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758414.933962:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11471 00000100:00080000:3.0:1787758414.933971:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758414.933977:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758414.934018:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655119616/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758414.934040:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758414.934043:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758414.934044:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758414.934049:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11471 00000100:00080000:3.0:1787758414.934052:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758414.934057:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758414.934067:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758414.934100:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655119616:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787758414.934101:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655119744/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758414.934118:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758414.934122:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758414.934123:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758414.934127:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11471 00000100:00080000:3.0:1787758414.934131:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758414.934134:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758414.934139:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758414.934157:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655119872/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787758414.934165:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758414.934176:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758414.934179:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758414.934180:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758414.934184:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11471 00000100:00080000:3.0:1787758414.934187:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758414.934189:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758414.934192:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758414.934200:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655120000/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758414.934206:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758414.934209:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758414.934210:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758414.934213:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11471 00000100:00080000:3.0:1787758414.934216:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758414.934220:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758414.934223:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758414.934257:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655120128/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758414.934264:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758414/real 1787758414] req@ffff8c05a9f78700 x1874588655119616/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758414.934266:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758414.934273:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758414.934276:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758414.934285:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11471 00000100:00080000:3.0:1787758414.934289:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758414.934309:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655119616/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758414.934313:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758414.934325:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787758414.934331:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758414.934337:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758414.934340:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655120256/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758414.934344:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655119616:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758414.934358:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758414.934360:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758414.934362:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758414.934365:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11471 00000100:00080000:3.0:1787758414.934367:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758414.934369:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758414.934373:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758414.934375:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655119744:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758414.934396:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655120384/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758414.934405:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758414.934424:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758414.934425:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758414.934428:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11471 00000100:00080000:3.0:1787758414.934432:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758414.934434:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758414.934439:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787758414.934440:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758414.934453:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655120512/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758414.934458:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758414/real 1787758414] req@ffff8c05a9f79880 x1874588655119744/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758414.934487:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655119744/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758414.934495:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758414.934498:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758414.934501:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655119744:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758414.934507:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655119872:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758414.934521:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758414.934528:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758414/real 1787758414] req@ffff8c05a9f7a300 x1874588655119872/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758414.934536:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655119872/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758414.934541:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758414.934544:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758414.934546:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655119872:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758414.934550:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655120000:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758414.934561:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758414.934566:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758414/real 1787758414] req@ffff8c05a9f7aa00 x1874588655120000/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758414.934574:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655120000/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758414.934579:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758414.934581:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758414.934582:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655120000:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758414.934585:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655120128:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758414.934595:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758414.934601:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758414/real 1787758414] req@ffff8c05a9f7a680 x1874588655120128/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758414.934608:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655120128/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758414.934613:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758414.934614:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758414.934616:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655120128:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758414.934627:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655120256:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758414.934639:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758414.934645:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758414/real 1787758414] req@ffff8c05a9f78a80 x1874588655120256/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758414.934668:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655120256/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758414.934674:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758414.934676:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758414.934678:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655120256:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758414.934691:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655120384:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758414.934703:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758414.934710:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758414/real 1787758414] req@ffff8c05a9f78380 x1874588655120384/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758414.934718:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655120384/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758414.934723:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758414.934724:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758414.934726:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655120384:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758414.934730:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655120512:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758414.934740:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758414.934745:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758414/real 1787758414] req@ffff8c05a9f7b480 x1874588655120512/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758414.934753:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655120512/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758414.934763:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758414.934765:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758414.934767:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655120512:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758415.061947:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758415.510050:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758416.085961:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758416.534124:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758417.110013:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758417.558120:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758418.133996:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758418.582201:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758419.158076:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758419.606189:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758420.054007:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758420.054020:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758420.054026:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758420.054039:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11476 00000100:00080000:3.0:1787758420.054053:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758420.054062:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758420.054109:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655120640/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758420.054145:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758420.054151:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758420.054154:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758420.054160:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11476 00000100:00080000:3.0:1787758420.054166:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758420.054171:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758420.054183:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758420.054202:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655120768/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758420.054225:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758420.054233:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758420.054238:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758420.054244:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655120640:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758420.054254:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11476 00000100:00080000:3.0:1787758420.054265:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758420.054274:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758420.054285:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758420.054354:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655120896/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758420.054371:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758420.054376:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758420.054380:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758420.054386:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11476 00000100:00080000:3.0:1787758420.054395:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787758420.054403:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758420.054409:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758420.054420:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758420.054437:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655121024/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758420.054449:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758420.054454:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758420.054457:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758420.054462:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11476 00000100:00080000:3.0:1787758420.054467:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758420.054473:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758420.054481:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758420/real 1787758420] req@ffff8c05a9f79500 x1874588655120640/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787758420.054484:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758420.054512:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655121152/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758420.054524:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655120640/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758420.054532:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758420.054548:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758420.054555:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758420.054563:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655120640:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758420.054577:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655120768:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758420.054633:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758420.054651:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758420.054659:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758420.054660:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758420/real 1787758420] req@ffff8c05a9f7b800 x1874588655120768/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758420.054677:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11476 00000100:00080000:3.0:1787758420.054689:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758420.054697:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655120768/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758420.054699:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:1.0:1787758420.054727:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758420.054732:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758420.054736:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655120768:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758420.054765:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655120896:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758420.054787:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758420.054801:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758420/real 1787758420] req@ffff8c05a9f78e00 x1874588655120896/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758420.054814:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655120896/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758420.054824:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758420.054828:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758420.054832:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655120896:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758420.054840:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655121024:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758420.054864:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758420.054937:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758420/real 1787758420] req@ffff8c05a9f7ad80 x1874588655121024/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758420.054961:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655121024/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787758420.054965:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787758420.054994:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787758420.055003:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655121280/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758420.055013:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758420.055036:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655121024:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758420.055055:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655121152:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758420.055101:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758420.055132:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758420.055135:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758420/real 1787758420] req@ffff8c05a9f7b100 x1874588655121152/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758420.055144:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758420.055153:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758420.055173:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11476 00000100:00080000:3.0:1787758420.055181:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758420.055193:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758420.055195:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655121152/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787758420.055203:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787758420.055224:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787758420.055233:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655121408/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758420.055237:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758420.055250:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655121152:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758420.055279:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758420.055286:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758420.055314:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758420.055315:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655121280:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758420.055347:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11476 00000100:00080000:3.0:1787758420.055358:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758420.055366:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758420.055376:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787758420.055392:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758420.055401:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655121536/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758420.055423:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758420/real 1787758420] req@ffff8c05a9f79f80 x1874588655121280/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758420.055450:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655121280/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758420.055461:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758420.055464:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758420.055468:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655121280:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758420.055476:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655121408:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758420.055502:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758420.055516:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758420/real 1787758420] req@ffff8c05a9f7bb80 x1874588655121408/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758420.055529:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655121408/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758420.055539:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758420.055542:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758420.055546:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655121408:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758420.055561:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655121536:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758420.055580:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758420.055590:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758420/real 1787758420] req@ffff8c05a9f79180 x1874588655121536/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758420.055604:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655121536/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758420.055613:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758420.055616:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758420.055619:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655121536:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758420.182003:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758420.630236:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758421.205997:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758421.654247:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758422.230002:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758422.678105:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758423.254039:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758423.702279:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758424.277975:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758424.726153:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758425.174021:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758425.174033:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758425.174038:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758425.174050:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11481 00000100:00080000:3.0:1787758425.174063:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758425.174071:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758425.174119:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655121664/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758425.174155:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758425.174160:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758425.174163:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758425.174168:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11481 00000100:00080000:3.0:1787758425.174172:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758425.174177:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758425.174189:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758425.174226:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655121792/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758425.174248:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655121664:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758425.174252:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758425.174264:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758425.174266:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758425.174272:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11481 00000100:00080000:3.0:1787758425.174276:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758425.174281:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758425.174287:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758425.174304:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655121920/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758425.174317:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758425.174321:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758425.174323:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758425.174328:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11481 00000100:00080000:3.0:1787758425.174332:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758425.174336:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:1.0:1787758425.174344:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:3.0:1787758425.174344:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758425.174364:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655122048/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758425.174384:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758425.174388:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758425.174391:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758425.174395:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11481 00000100:00080000:3.0:1787758425.174400:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758425.174405:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758425/real 1787758425] req@ffff8c05a9f79180 x1874588655121664/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758425.174411:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758425.174424:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758425.174443:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655121664/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758425.174454:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655122176/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758425.174465:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758425.174473:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758425.174478:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787758425.174481:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758425.174488:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758425.174491:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655121664:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758425.174498:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11481 00000100:00080000:3.0:1787758425.174507:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758425.174512:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758425.174517:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758425.174530:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655122304/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758425.174535:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655121792:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758425.174560:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758425.174565:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758425.174568:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758425.174573:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11481 00000100:00080000:3.0:1787758425.174579:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787758425.174582:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758425.174590:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758425.174605:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758425.174626:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758425/real 1787758425] req@ffff8c05a9f7bb80 x1874588655121792/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758425.174631:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655122432/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758425.174662:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655121792/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758425.174668:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758425.174679:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758425.174682:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758425.174687:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787758425.174692:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758425.174695:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655121792:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758425.174711:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11481 00000100:00100000:1.0:1787758425.174713:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655121920:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758425.174727:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758425.174737:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758425.174747:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787758425.174749:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758425.174777:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655122560/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758425.174778:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758425/real 1787758425] req@ffff8c05a9f79f80 x1874588655121920/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758425.174841:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655121920/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758425.174851:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758425.174855:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758425.174861:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655121920:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758425.174870:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655122048:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758425.174941:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758425.174964:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758425/real 1787758425] req@ffff8c05a9f7b100 x1874588655122048/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758425.174988:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655122048/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758425.175004:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758425.175010:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758425.175017:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655122048:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758425.175027:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655122176:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758425.175054:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758425.175068:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758425/real 1787758425] req@ffff8c05a9f7ad80 x1874588655122176/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758425.175082:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655122176/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758425.175092:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758425.175095:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758425.175099:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655122176:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758425.175119:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655122304:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758425.175155:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758425.175167:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758425/real 1787758425] req@ffff8c05a9f78e00 x1874588655122304/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758425.175182:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655122304/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758425.175193:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758425.175196:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758425.175200:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655122304:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758425.175237:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655122432:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758425.175263:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758425.175277:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758425/real 1787758425] req@ffff8c05a9f7b800 x1874588655122432/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758425.175292:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655122432/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758425.175302:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758425.175306:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758425.175310:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655122432:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758425.175317:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655122560:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758425.175334:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758425.175344:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758425/real 1787758425] req@ffff8c05a9f79500 x1874588655122560/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758425.175358:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655122560/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758425.175377:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758425.175380:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758425.175384:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655122560:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758425.302067:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758425.750267:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758426.326034:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758426.774140:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758427.350103:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758427.798294:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758428.374064:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758428.822238:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758429.398032:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758429.846177:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758430.294005:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758430.294021:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758430.294028:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758430.294042:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11486 00000100:00080000:3.0:1787758430.294056:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758430.294065:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758430.294133:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655122688/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758430.294173:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758430.294177:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758430.294180:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758430.294186:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11486 00000100:00080000:3.0:1787758430.294190:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758430.294195:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758430.294206:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758430.294225:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655122816/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758430.294238:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758430.294243:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758430.294245:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758430.294249:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11486 00000100:00080000:3.0:1787758430.294253:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758430.294257:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758430.294263:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758430.294273:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655122688:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787758430.294281:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655122944/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758430.294301:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758430.294305:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758430.294307:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758430.294312:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11486 00000100:00080000:3.0:1787758430.294316:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758430.294321:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758430.294326:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758430.294341:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655123072/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758430.294353:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758430.294359:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758430.294362:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758430.294369:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11486 00000100:00080000:3.0:1787758430.294376:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758430.294384:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:1.0:1787758430.294393:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:3.0:1787758430.294399:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758430.294429:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655123200/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758430.294448:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758430.294455:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758430.294457:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758430.294466:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11486 00000100:00100000:1.0:1787758430.294472:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758430/real 1787758430] req@ffff8c05a9f7b480 x1874588655122688/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758430.294474:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758430.294484:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758430.294495:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758430.294521:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655123328/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758430.294533:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758430.294537:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758430.294539:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758430.294545:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11486 00000100:00080000:3.0:1787758430.294548:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758430.294554:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655122688/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758430.294557:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758430.294567:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758430.294586:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655123456/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758430.294590:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758430.294608:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758430.294611:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787758430.294615:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758430.294623:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758430.294630:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655122688:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758430.294631:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11486 00000100:00080000:3.0:1787758430.294638:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758430.294649:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758430.294656:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758430.294663:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655122816:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758430.294671:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655123584/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787758430.294726:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758430.294753:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758430/real 1787758430] req@ffff8c05a9f78380 x1874588655122816/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758430.294778:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655122816/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758430.294822:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758430.294829:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758430.294836:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655122816:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758430.294921:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655122944:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758430.294963:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758430.294987:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758430/real 1787758430] req@ffff8c05a9f78a80 x1874588655122944/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758430.295011:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655122944/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758430.295027:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758430.295034:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758430.295040:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655122944:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758430.295052:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655123072:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758430.295085:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758430.295104:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758430/real 1787758430] req@ffff8c05a9f7a680 x1874588655123072/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758430.295158:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655123072/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758430.295174:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758430.295180:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758430.295188:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655123072:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758430.295199:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655123200:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758430.295236:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758430.295256:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758430/real 1787758430] req@ffff8c05a9f7aa00 x1874588655123200/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758430.295278:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655123200/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758430.295293:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758430.295299:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758430.295306:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655123200:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758430.295318:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655123328:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758430.295353:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758430.295373:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758430/real 1787758430] req@ffff8c05a9f7a300 x1874588655123328/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758430.295395:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655123328/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758430.295410:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758430.295415:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758430.295421:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655123328:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758430.295433:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655123456:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758430.295465:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758430.295485:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758430/real 1787758430] req@ffff8c05a9f79880 x1874588655123456/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758430.295505:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655123456/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758430.295519:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758430.295524:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758430.295531:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655123456:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758430.295541:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655123584:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758430.295572:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758430.295591:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758430/real 1787758430] req@ffff8c05a9f78700 x1874588655123584/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758430.295611:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655123584/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758430.295626:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758430.295632:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758430.295638:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655123584:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758430.422020:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758430.870158:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758431.446030:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758431.894263:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758432.470006:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758432.918214:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758433.493986:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758433.942307:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758434.518015:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758434.966159:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758435.414013:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758435.414041:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787758435.414045:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758435.414053:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11491 00000100:00080000:2.0:1787758435.414063:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758435.414069:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758435.414101:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588655123712/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758435.414132:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758435.414135:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758435.414136:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758435.414139:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11491 00000100:00080000:2.0:1787758435.414142:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758435.414144:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758435.414151:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758435.414161:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588655123840/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758435.414172:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758435.414175:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758435.414177:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758435.414180:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11491 00000100:00080000:2.0:1787758435.414183:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758435.414191:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655123712:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787758435.414191:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758435.414201:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758435.414209:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588655123968/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758435.414216:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758435.414219:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758435.414220:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758435.414224:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11491 00000100:00080000:2.0:1787758435.414226:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758435.414229:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758435.414233:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758435.414245:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588655124096/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758435.414252:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758435.414255:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758435.414256:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758435.414259:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11491 00000100:00080000:2.0:1787758435.414261:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758435.414263:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758435.414266:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758435.414272:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588655124224/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758435.414278:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758435.414281:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758435.414285:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787758435.414287:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787758435.414291:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11491 00000100:00080000:2.0:1787758435.414294:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758435.414306:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758435.414309:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758435.414324:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588655124352/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758435.414331:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758435.414334:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787758435.414337:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758435.414340:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11491 00000100:00080000:2.0:1787758435.414343:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758435.414346:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758435.414349:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758435.414357:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588655124480/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758435.414375:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758435.414377:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758435.414379:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758435.414382:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11491 00000100:00080000:2.0:1787758435.414385:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758435.414388:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758435.414391:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758435.414398:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758435/real 1787758435] req@ffff8c05b2e70e00 x1874588655123712/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758435.414404:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588655124608/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758435.414476:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588655123712/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758435.414487:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758435.414491:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758435.414495:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655123712:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758435.414538:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655123840:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758435.414559:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758435.414568:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758435/real 1787758435] req@ffff8c05b2e72300 x1874588655123840/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758435.414577:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588655123840/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758435.414583:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758435.414586:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758435.414588:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655123840:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758435.414592:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655123968:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758435.414603:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758435.414609:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758435/real 1787758435] req@ffff8c05b2e72a00 x1874588655123968/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758435.414616:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588655123968/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758435.414626:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758435.414628:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758435.414630:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655123968:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758435.414634:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655124096:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758435.414645:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758435.414651:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758435/real 1787758435] req@ffff8c05b2e72680 x1874588655124096/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758435.414658:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588655124096/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758435.414664:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758435.414666:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758435.414668:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655124096:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758435.414671:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655124224:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758435.414682:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758435.414688:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758435/real 1787758435] req@ffff8c05b2e71180 x1874588655124224/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758435.414695:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588655124224/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758435.414700:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758435.414702:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758435.414705:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655124224:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758435.414709:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655124352:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758435.414720:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758435.414726:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758435/real 1787758435] req@ffff8c05b2e70000 x1874588655124352/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758435.414733:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588655124352/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758435.414738:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758435.414740:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758435.414741:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655124352:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758435.414745:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655124480:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758435.414755:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758435.414761:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758435/real 1787758435] req@ffff8c05b2e70a80 x1874588655124480/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758435.414767:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588655124480/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758435.414772:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758435.414774:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758435.414776:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655124480:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758435.414779:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655124608:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758435.414789:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758435.414796:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758435/real 1787758435] req@ffff8c05b2e73100 x1874588655124608/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758435.414803:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588655124608/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758435.414808:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758435.414810:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758435.414812:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655124608:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758435.541963:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758435.990169:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758436.565960:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758437.014059:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758437.590008:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758438.038200:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758438.614042:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758439.062159:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758439.638003:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758440.086177:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758440.534095:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758440.534103:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758440.534107:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758440.534115:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11496 00000100:00080000:3.0:1787758440.534125:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758440.534131:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758440.534165:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655124736/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758440.534193:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758440.534196:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758440.534197:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758440.534200:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11496 00000100:00080000:3.0:1787758440.534203:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758440.534205:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758440.534213:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758440.534222:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655124864/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758440.534228:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758440.534231:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758440.534233:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758440.534236:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11496 00000100:00080000:3.0:1787758440.534240:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758440.534244:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758440.534259:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758440.534267:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655124992/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758440.534274:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758440.534277:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758440.534279:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758440.534282:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11496 00000100:00080000:3.0:1787758440.534284:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758440.534288:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758440.534291:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758440.534301:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655125120/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758440.534307:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655124736:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758440.534311:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758440.534331:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758440.534334:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758440.534336:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11496 00000100:00080000:3.0:1787758440.534338:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758440.534341:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758440.534345:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758440.534355:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655125248/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758440.534362:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758440.534365:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758440.534366:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758440.534370:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11496 00000100:00080000:3.0:1787758440.534372:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758440.534374:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758440.534380:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758440.534389:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655125376/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758440.534397:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758440.534401:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758440.534402:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758440.534404:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11496 00000100:00080000:3.0:1787758440.534406:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758440.534408:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758440.534412:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758440.534420:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655125504/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758440.534426:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758440.534428:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758440.534430:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758440.534438:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11496 00000800:00000100:1.0:1787758440.534442:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758440.534445:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758440.534451:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758440.534457:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758440.534473:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655125632/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758440.534515:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758440/real 1787758440] req@ffff8c05a9f78700 x1874588655124736/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758440.534554:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655124736/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758440.534576:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758440.534585:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758440.534595:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655124736:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758440.534627:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655124864:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758440.534664:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758440.534682:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758440/real 1787758440] req@ffff8c05a9f79880 x1874588655124864/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758440.534705:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655124864/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758440.534716:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758440.534720:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758440.534724:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655124864:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758440.534731:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655124992:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758440.534771:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758440.534785:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758440/real 1787758440] req@ffff8c05a9f7a300 x1874588655124992/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758440.534799:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655124992/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758440.534808:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758440.534812:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758440.534817:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655124992:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758440.534853:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655125120:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758440.534956:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758440.534996:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758440/real 1787758440] req@ffff8c05a9f7aa00 x1874588655125120/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758440.535011:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655125120/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758440.535029:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758440.535033:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758440.535037:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655125120:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758440.535044:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655125248:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758440.535071:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758440.535083:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758440/real 1787758440] req@ffff8c05a9f7a680 x1874588655125248/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758440.535096:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655125248/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758440.535105:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758440.535108:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758440.535112:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655125248:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758440.535118:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655125376:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758440.535136:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758440.535147:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758440/real 1787758440] req@ffff8c05a9f78a80 x1874588655125376/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758440.535161:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655125376/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758440.535169:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758440.535172:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758440.535176:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655125376:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758440.535183:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655125504:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758440.535201:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758440.535211:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758440/real 1787758440] req@ffff8c05a9f78380 x1874588655125504/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758440.535224:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655125504/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758440.535233:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758440.535236:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758440.535240:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655125504:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758440.535263:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655125632:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758440.535285:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758440.535296:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758440/real 1787758440] req@ffff8c05a9f7b480 x1874588655125632/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758440.535310:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655125632/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758440.535320:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758440.535323:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758440.535327:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655125632:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758440.662036:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758441.110111:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758441.686014:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758442.134149:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758442.710029:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758443.158037:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758443.734007:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758444.182099:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758444.757990:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758445.206038:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758445.653989:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758445.653998:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758445.654001:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758445.654010:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11502 00000100:00080000:1.0:1787758445.654020:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758445.654027:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758445.654060:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655125760/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758445.654098:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758445.654101:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758445.654102:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758445.654105:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11502 00000100:00080000:1.0:1787758445.654107:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758445.654109:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758445.654117:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758445.654126:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655125888/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758445.654133:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758445.654142:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758445.654143:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758445.654148:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11502 00000100:00100000:0.0:1787758445.654150:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655125760:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758445.654151:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758445.654158:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758445.654163:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758445.654174:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655126016/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758445.654182:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758445.654185:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758445.654186:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758445.654189:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11502 00000100:00080000:1.0:1787758445.654206:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758445.654209:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758445.654212:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787758445.654216:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758445.654221:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655126144/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758445.654230:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758445.654232:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758445.654234:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758445.654236:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11502 00000100:00080000:1.0:1787758445.654238:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758445.654241:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758445.654244:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758445.654251:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655126272/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758445.654257:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787758445.654259:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758445/real 1787758445] req@ffff8c05b2d9b480 x1874588655125760/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758445.654261:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758445.654264:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758445.654273:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11502 00000100:00080000:1.0:1787758445.654278:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758445.654295:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758445.654299:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758445.654304:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655125760/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758445.654309:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655126400/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758445.654321:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758445.654323:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758445.654324:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758445.654326:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11502 00000100:00080000:1.0:1787758445.654328:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758445.654329:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758445.654332:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787758445.654336:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000008:00100000:1.0:1787758445.654337:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758445.654342:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655125760:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758445.654346:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655126528/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758445.654350:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655125888:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758445.654362:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758445.654364:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758445.654366:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758445.654370:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11502 00000100:00080000:1.0:1787758445.654376:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758445.654381:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:0.0:1787758445.654383:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:1.0:1787758445.654386:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758445.654396:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655126656/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758445.654404:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758445/real 1787758445] req@ffff8c05b2d98000 x1874588655125888/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758445.654422:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655125888/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758445.654432:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758445.654435:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758445.654437:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655125888:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758445.654459:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655126016:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758445.654476:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758445.654484:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758445/real 1787758445] req@ffff8c05b2d99f80 x1874588655126016/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758445.654492:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655126016/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758445.654501:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758445.654504:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758445.654506:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655126016:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758445.654509:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655126144:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758445.654519:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758445.654525:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758445/real 1787758445] req@ffff8c05b2d99880 x1874588655126144/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758445.654542:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655126144/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758445.654547:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758445.654549:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758445.654551:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655126144:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758445.654555:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655126272:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758445.654564:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758445.654570:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758445/real 1787758445] req@ffff8c05b2d9b100 x1874588655126272/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758445.654577:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655126272/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758445.654582:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758445.654584:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758445.654587:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655126272:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758445.654590:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655126400:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758445.654613:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758445.654618:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758445/real 1787758445] req@ffff8c05b2d98700 x1874588655126400/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758445.654625:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655126400/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758445.654630:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758445.654632:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758445.654634:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655126400:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758445.654637:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655126528:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758445.654646:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758445.654652:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758445/real 1787758445] req@ffff8c05b2d9bb80 x1874588655126528/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758445.654659:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655126528/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758445.654664:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758445.654666:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758445.654667:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655126528:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758445.654671:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655126656:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758445.654680:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758445.654686:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758445/real 1787758445] req@ffff8c05b2d9aa00 x1874588655126656/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758445.654693:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655126656/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758445.654698:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758445.654699:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758445.654701:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655126656:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758445.781947:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758446.230065:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758446.805933:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758447.254044:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758447.829956:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758448.278074:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758448.853946:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758449.302059:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758449.877937:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758450.326075:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758450.773970:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758450.773978:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758450.773981:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758450.773990:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11507 00000100:00080000:3.0:1787758450.773999:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758450.774005:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758450.774053:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655126784/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758450.774090:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758450.774094:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758450.774096:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758450.774101:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11507 00000100:00080000:3.0:1787758450.774106:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758450.774109:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758450.774131:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758450.774142:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655126912/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758450.774150:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758450.774153:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758450.774154:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758450.774157:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11507 00000100:00080000:3.0:1787758450.774159:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758450.774162:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758450.774165:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758450.774168:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655126784:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787758450.774175:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655127040/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758450.774184:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758450.774186:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758450.774187:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758450.774190:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11507 00000100:00080000:3.0:1787758450.774193:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758450.774195:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758450.774199:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758450.774207:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655127168/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758450.774213:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758450.774216:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758450.774217:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758450.774220:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11507 00000100:00080000:3.0:1787758450.774223:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758450.774225:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758450.774228:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758450.774239:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655127296/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787758450.774240:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758450.774249:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758450.774252:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758450.774253:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758450.774256:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11507 00000100:00080000:3.0:1787758450.774258:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758450.774261:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758450.774264:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758450.774272:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655127424/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758450.774281:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758450.774284:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758450.774285:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787758450.774290:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758450/real 1787758450] req@ffff8c05a9f79500 x1874588655126784/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758450.774290:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11507 00000100:00080000:3.0:1787758450.774292:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758450.774298:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758450.774306:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758450.774316:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655127552/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758450.774319:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655126784/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758450.774336:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758450.774340:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758450.774340:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758450.774342:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758450.774346:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787758450.774350:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11507 00000100:00100000:0.0:1787758450.774354:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655126784:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758450.774355:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758450.774363:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758450.774365:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655126912:192.168.201.107@tcp:38:kworker.0 00000008:00100000:3.0:1787758450.774371:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758450.774389:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655127680/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787758450.774393:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758450.774406:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758450/real 1787758450] req@ffff8c05a9f7b800 x1874588655126912/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758450.774416:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655126912/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758450.774424:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758450.774428:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758450.774432:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655126912:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758450.774467:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655127040:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758450.774487:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758450.774496:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758450/real 1787758450] req@ffff8c05a9f78e00 x1874588655127040/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758450.774504:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655127040/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758450.774509:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758450.774512:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758450.774529:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655127040:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758450.774534:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655127168:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758450.774547:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758450.774554:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758450/real 1787758450] req@ffff8c05a9f7ad80 x1874588655127168/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758450.774561:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655127168/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758450.774567:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758450.774568:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758450.774571:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655127168:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758450.774575:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655127296:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758450.774585:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758450.774591:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758450/real 1787758450] req@ffff8c05a9f7b100 x1874588655127296/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758450.774598:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655127296/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758450.774604:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758450.774606:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758450.774608:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655127296:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758450.774611:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655127424:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758450.774620:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758450.774626:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758450/real 1787758450] req@ffff8c05a9f79f80 x1874588655127424/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758450.774633:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655127424/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758450.774639:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758450.774641:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758450.774643:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655127424:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758450.774647:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655127552:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758450.774657:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758450.774680:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758450/real 1787758450] req@ffff8c05a9f7bb80 x1874588655127552/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758450.774693:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655127552/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758450.774701:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758450.774704:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758450.774707:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655127552:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758450.774713:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655127680:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758450.774745:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758450.774768:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758450/real 1787758450] req@ffff8c05a9f79180 x1874588655127680/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758450.774777:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655127680/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758450.774783:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758450.774785:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758450.774787:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655127680:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758450.901959:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758451.350001:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758451.925992:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758452.374075:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758452.949979:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758453.398061:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758453.974035:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758454.422068:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758454.997981:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758455.446025:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758455.893969:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758455.893978:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758455.893981:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758455.893990:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11512 00000100:00080000:3.0:1787758455.893999:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758455.894005:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758455.894039:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655127808/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758455.894062:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758455.894065:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758455.894066:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758455.894069:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11512 00000100:00080000:3.0:1787758455.894071:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758455.894073:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758455.894080:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758455.894090:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655127936/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758455.894097:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758455.894106:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758455.894107:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758455.894113:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11512 00000100:00100000:0.0:1787758455.894115:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655127808:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758455.894116:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758455.894122:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758455.894126:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758455.894134:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655128064/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758455.894142:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758455.894144:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758455.894146:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758455.894148:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11512 00000100:00080000:3.0:1787758455.894151:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758455.894153:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758455.894156:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758455.894163:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655128192/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758455.894169:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758455.894172:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758455.894173:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758455.894174:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11512 00000100:00080000:3.0:1787758455.894177:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758455.894182:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:0.0:1787758455.894185:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:3.0:1787758455.894187:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758455.894196:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655128320/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758455.894203:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758455.894205:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758455.894207:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758455.894209:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11512 00000100:00080000:3.0:1787758455.894212:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758455.894214:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758455.894218:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758455.894225:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655128448/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758455.894230:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758455.894233:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758455.894234:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758455.894237:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11512 00000100:00080000:3.0:1787758455.894239:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758455.894244:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758455.894245:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758455/real 1787758455] req@ffff8c05a9f79180 x1874588655127808/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787758455.894247:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758455.894259:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655128576/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758455.894266:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758455.894269:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758455.894270:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787758455.894275:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655127808/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758455.894276:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11512 00000100:00080000:3.0:1787758455.894283:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758455.894291:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787758455.894298:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787758455.894300:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787758455.894308:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758455.894317:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655127808:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787758455.894318:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655128704/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758455.894338:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655127936:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758455.894367:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758455.894383:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758455/real 1787758455] req@ffff8c05a9f7bb80 x1874588655127936/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758455.894397:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655127936/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758455.894409:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758455.894413:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758455.894418:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655127936:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758455.894445:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655128064:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758455.894464:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758455.894474:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758455/real 1787758455] req@ffff8c05a9f79f80 x1874588655128064/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758455.894483:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655128064/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758455.894489:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758455.894492:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758455.894495:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655128064:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758455.894504:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655128192:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758455.894522:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758455.894530:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758455/real 1787758455] req@ffff8c05a9f7b100 x1874588655128192/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758455.894542:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655128192/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758455.894549:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758455.894552:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758455.894555:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655128192:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758455.894560:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655128320:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758455.894576:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758455.894587:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758455/real 1787758455] req@ffff8c05a9f7ad80 x1874588655128320/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758455.894594:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655128320/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758455.894600:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758455.894602:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758455.894604:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655128320:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758455.894607:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655128448:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758455.894616:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758455.894622:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758455/real 1787758455] req@ffff8c05a9f78e00 x1874588655128448/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758455.894630:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655128448/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758455.894635:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758455.894637:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758455.894640:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655128448:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758455.894646:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655128576:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758455.894676:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758455.894686:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758455/real 1787758455] req@ffff8c05a9f7b800 x1874588655128576/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758455.894695:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655128576/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758455.894701:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758455.894704:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758455.894708:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655128576:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758455.894714:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655128704:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758455.894730:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758455.894741:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758455/real 1787758455] req@ffff8c05a9f79500 x1874588655128704/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758455.894749:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655128704/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758455.894759:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758455.894761:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758455.894763:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655128704:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758456.021942:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758456.470196:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758457.045990:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758457.494016:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758458.069929:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758458.518216:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758459.093997:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758459.542037:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758460.118016:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758460.566142:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758461.013957:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758461.013968:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758461.013974:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758461.013984:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11517 00000100:00080000:3.0:1787758461.013996:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758461.014003:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758461.014048:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655128832/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758461.014076:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758461.014079:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758461.014080:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758461.014083:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11517 00000100:00080000:3.0:1787758461.014087:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758461.014090:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758461.014099:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758461.014109:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655128960/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758461.014117:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655128832:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758461.014120:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758461.014127:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758461.014128:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758461.014132:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11517 00000100:00080000:3.0:1787758461.014134:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758461.014137:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758461.014140:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758461.014149:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655129088/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758461.014157:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758461.014160:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758461.014161:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758461.014164:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11517 00000100:00080000:3.0:1787758461.014166:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758461.014168:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758461.014172:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758461.014178:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655129216/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758461.014184:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758461.014187:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758461.014189:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758461.014191:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11517 00000100:00080000:3.0:1787758461.014195:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787758461.014198:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758461.014202:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758461.014209:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758461.014222:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655129344/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758461.014233:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758461.014237:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758461.014240:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758461.014245:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11517 00000100:00100000:0.0:1787758461.014247:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758461/real 1787758461] req@ffff8c05a9f7b480 x1874588655128832/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758461.014251:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758461.014261:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758461.014266:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758461.014274:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655128832/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758461.014278:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655129472/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758461.014292:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758461.014300:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758461.014302:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787758461.014304:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758461.014308:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787758461.014312:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655128832:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758461.014313:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11517 00000100:00080000:3.0:1787758461.014319:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758461.014323:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758461.014327:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758461.014335:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655129600/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758461.014341:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758461.014344:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787758461.014347:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655128960:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758461.014348:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758461.014355:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11517 00000100:00080000:3.0:1787758461.014358:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758461.014360:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758461.014364:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758461.014374:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655129728/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787758461.014382:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758461.014395:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758461/real 1787758461] req@ffff8c05a9f78380 x1874588655128960/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758461.014405:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655128960/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758461.014412:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758461.014414:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758461.014417:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655128960:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758461.014422:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655129088:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758461.014434:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758461.014441:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758461/real 1787758461] req@ffff8c05a9f78a80 x1874588655129088/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758461.014448:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655129088/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758461.014453:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758461.014455:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758461.014457:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655129088:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758461.014461:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655129216:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758461.014473:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758461.014479:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758461/real 1787758461] req@ffff8c05a9f7a680 x1874588655129216/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758461.014487:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655129216/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758461.014492:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758461.014493:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758461.014505:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655129216:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758461.014508:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655129344:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758461.014521:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758461.014528:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758461/real 1787758461] req@ffff8c05a9f7aa00 x1874588655129344/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758461.014536:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655129344/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758461.014542:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758461.014543:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758461.014545:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655129344:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758461.014548:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655129472:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758461.014559:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758461.014565:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758461/real 1787758461] req@ffff8c05a9f7a300 x1874588655129472/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758461.014587:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655129472/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758461.014594:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758461.014596:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758461.014598:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655129472:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758461.014603:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655129600:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758461.014618:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758461.014626:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758461/real 1787758461] req@ffff8c05a9f79880 x1874588655129600/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758461.014640:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655129600/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758461.014646:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758461.014649:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758461.014651:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655129600:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758461.014666:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655129728:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758461.014680:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758461.014687:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758461/real 1787758461] req@ffff8c05a9f78700 x1874588655129728/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758461.014695:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655129728/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758461.014700:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758461.014702:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758461.014704:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655129728:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758461.141955:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758461.590091:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758462.165972:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758462.614148:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758463.189990:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758463.638055:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758464.213976:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758464.662129:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758465.237957:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758465.686013:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758466.133939:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758466.133947:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758466.133950:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758466.133958:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11522 00000100:00080000:3.0:1787758466.133966:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758466.133972:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758466.134003:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655129856/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758466.134026:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758466.134029:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758466.134030:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758466.134033:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11522 00000100:00080000:3.0:1787758466.134035:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758466.134038:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758466.134045:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758466.134054:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655129984/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758466.134061:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758466.134063:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758466.134065:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758466.134068:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11522 00000100:00080000:3.0:1787758466.134070:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758466.134072:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758466.134074:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758466.134080:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655130112/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758466.134087:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758466.134089:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758466.134090:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758466.134094:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11522 00000100:00080000:3.0:1787758466.134096:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758466.134098:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758466.134101:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758466.134107:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655130240/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758466.134113:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655129856:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758466.134117:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758466.134120:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758466.134121:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758466.134124:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11522 00000100:00080000:3.0:1787758466.134126:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758466.134129:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758466.134132:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758466.134141:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655130368/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758466.134148:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758466.134150:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758466.134152:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758466.134154:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11522 00000100:00080000:3.0:1787758466.134157:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758466.134159:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758466.134161:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758466.134170:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655130496/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758466.134176:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758466.134179:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758466.134180:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758466.134182:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11522 00000100:00080000:3.0:1787758466.134184:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758466.134186:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758466.134188:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758466.134194:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655130624/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758466.134199:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758466.134201:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758466.134202:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758466.134205:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11522 00000100:00080000:3.0:1787758466.134206:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758466.134209:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758466.134213:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758466.134225:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655130752/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758466.134226:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655129984:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758466.134274:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655130112:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758466.134303:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655130240:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758466.134323:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655130368:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758466.134344:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655130496:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758466.134366:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655130624:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758466.134387:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655130752:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758466.261998:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758466.710079:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758467.285944:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758467.734049:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758468.309994:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758468.758161:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787758469.206146:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.107:988 00000400:00000100:1.0:1787758469.206185:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.107@tcp at host 192.168.201.107 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787758469.206206:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787758469.206247:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787758469.206258:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787758469.206267:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787758469.206278:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787758469.206289:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787758469.206300:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787758469.206313:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:0.0:1787758469.206315:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758466/real 1787758469] req@ffff8c05a9f7aa00 x1874588655130240/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758469.206345:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655130240/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758469.206365:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758469.206370:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758469.206375:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655130240:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758469.206387:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758466/real 1787758469] req@ffff8c05a9f7a680 x1874588655130368/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758469.206400:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655130368/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758469.206408:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758469.206411:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758469.206415:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655130368:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758469.206422:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758466/real 1787758469] req@ffff8c05a9f78a80 x1874588655130496/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758469.206452:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655130496/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758469.206460:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758469.206464:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758469.206467:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655130496:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758469.206474:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758466/real 1787758469] req@ffff8c05a9f78380 x1874588655130624/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758469.206486:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655130624/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758469.206494:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758469.206498:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758469.206501:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655130624:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758469.206508:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758466/real 1787758469] req@ffff8c05a9f7b480 x1874588655130752/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758469.206519:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655130752/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758469.206527:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758469.206530:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758469.206534:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655130752:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758469.206541:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758466/real 1787758469] req@ffff8c05a9f78700 x1874588655129856/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758469.206553:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655129856/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758469.206560:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758469.206564:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758469.206567:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655129856:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787758469.206573:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758466/real 1787758469] req@ffff8c05a9f79880 x1874588655129984/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758469.206585:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655129984/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758469.206593:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758469.206597:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758469.206600:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655129984:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758469.206607:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758466/real 1787758469] req@ffff8c05a9f7a300 x1874588655130112/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758469.206620:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655130112/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758469.206627:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758469.206631:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758469.206634:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655130112:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758469.333970:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758469.782051:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758470.357977:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758470.806033:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758471.254042:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758471.254052:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787758471.254065:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758471.254075:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11527 00000100:00080000:2.0:1787758471.254086:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758471.254094:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758471.254133:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588655130880/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758471.254168:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758471.254172:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758471.254174:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758471.254179:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11527 00000100:00080000:2.0:1787758471.254183:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758471.254188:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758471.254199:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758471.254219:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588655131008/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758471.254220:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655130880:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787758471.254245:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758471.254249:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758471.254252:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758471.254257:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11527 00000100:00080000:2.0:1787758471.254261:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758471.254265:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758471.254272:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758471.254285:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588655131136/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758471.254297:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787758471.254302:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787758471.254303:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758471.254308:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758471.254313:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11527 00000100:00080000:2.0:1787758471.254318:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758471.254323:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758471.254329:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758471.254341:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588655131264/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758471.254351:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758471.254354:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758471.254356:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758471.254362:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11527 00000100:00100000:0.0:1787758471.254363:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758471/real 1787758471] req@ffff8c05b2e72d80 x1874588655130880/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758471.254366:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758471.254375:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758471.254383:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758471.254406:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588655130880/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758471.254418:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588655131392/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758471.254422:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758471.254434:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787758471.254439:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787758471.254442:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655130880:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787758471.254448:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758471.254453:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758471.254459:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11527 00000100:00080000:2.0:1787758471.254464:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758471.254469:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758471.254474:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758471.254484:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655131008:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758471.254491:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588655131520/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758471.254505:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758471.254509:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787758471.254511:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758471.254516:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11527 00000800:00000100:0.0:1787758471.254520:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787758471.254524:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758471.254531:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758471.254537:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758471.254541:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758471/real 1787758471] req@ffff8c05b2e73100 x1874588655131008/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758471.254553:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588655131648/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758471.254560:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588655131008/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758471.254573:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758471.254579:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758471.254579:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758471.254583:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758471.254587:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787758471.254593:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11527 00000100:00100000:0.0:1787758471.254595:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655131008:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787758471.254600:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787758471.254606:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655131136:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787758471.254609:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758471.254618:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758471.254633:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588655131776/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787758471.254646:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758471.254661:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758471/real 1787758471] req@ffff8c05b2e70a80 x1874588655131136/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758471.254673:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588655131136/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758471.254687:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758471.254691:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758471.254694:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655131136:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758471.254700:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655131264:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758471.254720:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758471.254730:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758471/real 1787758471] req@ffff8c05b2e70000 x1874588655131264/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758471.254741:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588655131264/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758471.254749:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758471.254753:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758471.254756:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655131264:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758471.254763:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655131392:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758471.254781:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758471.254793:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758471/real 1787758471] req@ffff8c05b2e71180 x1874588655131392/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758471.254803:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588655131392/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758471.254812:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758471.254816:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758471.254820:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655131392:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758471.254839:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655131520:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758471.254859:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758471.254870:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758471/real 1787758471] req@ffff8c05b2e72680 x1874588655131520/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758471.254969:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588655131520/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758471.254978:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758471.254982:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758471.254985:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655131520:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758471.255019:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655131648:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758471.255044:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758471.255057:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758471/real 1787758471] req@ffff8c05b2e72a00 x1874588655131648/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758471.255070:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588655131648/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758471.255078:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758471.255082:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758471.255085:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655131648:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758471.255093:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655131776:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758471.255115:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758471.255128:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758471/real 1787758471] req@ffff8c05b2e72300 x1874588655131776/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758471.255141:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588655131776/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758471.255149:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758471.255152:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758471.255156:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655131776:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758471.382000:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758471.830071:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758472.406004:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758472.854119:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758473.429985:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758473.878070:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758474.453994:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758474.902075:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758475.477969:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758475.926016:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758476.374045:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758476.374056:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758476.374059:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758476.374068:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11532 00000100:00080000:3.0:1787758476.374078:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758476.374084:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758476.374117:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655131904/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758476.374140:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758476.374143:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758476.374144:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758476.374148:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11532 00000100:00080000:3.0:1787758476.374154:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758476.374157:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758476.374164:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758476.374172:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655132032/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758476.374180:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758476.374183:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758476.374184:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758476.374188:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11532 00000100:00080000:3.0:1787758476.374190:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758476.374193:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758476.374196:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758476.374203:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655132160/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758476.374211:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655131904:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758476.374214:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758476.374222:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758476.374223:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758476.374226:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11532 00000100:00080000:3.0:1787758476.374228:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758476.374230:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758476.374233:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758476.374241:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655132288/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758476.374247:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758476.374250:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758476.374251:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758476.374253:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11532 00000100:00080000:3.0:1787758476.374256:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758476.374258:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758476.374262:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758476.374270:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655132416/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758476.374282:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758476.374284:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758476.374286:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787758476.374290:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758476.374294:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11532 00000100:00080000:3.0:1787758476.374300:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758476.374319:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758476.374325:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758476.374339:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655132544/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758476.374349:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758476.374351:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758476.374353:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758476.374357:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11532 00000100:00100000:0.0:1787758476.374359:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758476/real 1787758476] req@ffff8c05a9f79500 x1874588655131904/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758476.374361:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758476.374368:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758476.374376:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758476.374388:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655132672/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758476.374391:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655131904/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758476.374423:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758476.374424:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758476.374428:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758476.374431:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758476.374434:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787758476.374437:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11532 00000100:00080000:3.0:1787758476.374441:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787758476.374444:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655131904:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758476.374446:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758476.374453:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758476.374461:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655132032:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758476.374464:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655132800/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787758476.374494:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758476.374511:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758476/real 1787758476] req@ffff8c05a9f7b800 x1874588655132032/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758476.374525:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655132032/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758476.374535:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758476.374539:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758476.374543:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655132032:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758476.374586:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655132160:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758476.374610:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758476.374623:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758476/real 1787758476] req@ffff8c05a9f78e00 x1874588655132160/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758476.374636:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655132160/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758476.374644:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758476.374648:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758476.374651:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655132160:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758476.374656:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655132288:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758476.374672:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758476.374679:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758476/real 1787758476] req@ffff8c05a9f7ad80 x1874588655132288/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758476.374687:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655132288/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758476.374692:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758476.374694:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758476.374697:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655132288:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758476.374700:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655132416:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758476.374711:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758476.374716:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758476/real 1787758476] req@ffff8c05a9f7b100 x1874588655132416/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758476.374724:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655132416/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758476.374729:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758476.374730:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758476.374732:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655132416:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758476.374735:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655132544:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758476.374745:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758476.374750:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758476/real 1787758476] req@ffff8c05a9f79f80 x1874588655132544/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758476.374757:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655132544/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758476.374762:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758476.374764:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758476.374766:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655132544:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758476.374769:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655132672:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758476.374778:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758476.374784:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758476/real 1787758476] req@ffff8c05a9f7bb80 x1874588655132672/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758476.374792:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655132672/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758476.374797:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758476.374798:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758476.374800:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655132672:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758476.374803:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655132800:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758476.374812:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758476.374817:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758476/real 1787758476] req@ffff8c05a9f79180 x1874588655132800/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758476.374824:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655132800/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758476.374829:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758476.374831:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758476.374841:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655132800:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758476.501999:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758476.950110:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758477.525998:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758477.974100:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758478.549966:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758478.998102:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758479.573967:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758480.022046:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758480.597974:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758481.046064:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758481.493939:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758481.493951:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758481.493955:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758481.493966:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11537 00000100:00080000:3.0:1787758481.493977:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758481.493984:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758481.494016:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655132928/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758481.494040:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758481.494043:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758481.494044:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758481.494048:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11537 00000100:00080000:3.0:1787758481.494050:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758481.494054:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758481.494062:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758481.494069:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655132928:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787758481.494074:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655133056/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758481.494083:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758481.494085:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758481.494086:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758481.494089:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11537 00000100:00080000:3.0:1787758481.494091:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758481.494093:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758481.494096:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758481.494105:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655133184/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758481.494112:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758481.494114:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758481.494115:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758481.494117:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11537 00000100:00080000:3.0:1787758481.494121:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758481.494123:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758481.494126:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787758481.494134:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758481.494135:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655133312/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758481.494144:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758481.494146:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758481.494148:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758481.494150:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11537 00000100:00080000:3.0:1787758481.494153:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758481.494155:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758481.494158:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758481.494165:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655133440/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758481.494171:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758481.494173:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758481.494174:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758481.494177:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11537 00000100:00080000:3.0:1787758481.494180:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758481.494185:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758481.494186:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758481/real 1787758481] req@ffff8c05a9f79180 x1874588655132928/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787758481.494187:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758481.494200:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655133568/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758481.494234:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758481.494236:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758481.494238:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787758481.494242:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655132928/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758481.494251:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11537 00000100:00080000:3.0:1787758481.494257:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758481.494259:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758481.494261:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787758481.494267:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000008:00100000:3.0:1787758481.494268:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758481.494271:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655132928:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787758481.494281:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655133696/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758481.494288:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758481.494291:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758481.494292:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787758481.494296:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655133056:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758481.494297:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11537 00000100:00080000:3.0:1787758481.494302:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758481.494305:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758481.494308:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758481.494317:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655133824/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787758481.494326:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758481.494338:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758481/real 1787758481] req@ffff8c05a9f7bb80 x1874588655133056/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758481.494347:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655133056/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758481.494353:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758481.494355:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758481.494357:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655133056:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758481.494361:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655133184:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758481.494371:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758481.494378:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758481/real 1787758481] req@ffff8c05a9f79f80 x1874588655133184/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758481.494385:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655133184/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758481.494394:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758481.494396:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758481.494398:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655133184:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758481.494401:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655133312:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758481.494411:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758481.494417:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758481/real 1787758481] req@ffff8c05a9f7b100 x1874588655133312/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758481.494424:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655133312/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758481.494429:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758481.494431:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758481.494433:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655133312:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758481.494437:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655133440:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758481.494446:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758481.494452:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758481/real 1787758481] req@ffff8c05a9f7ad80 x1874588655133440/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758481.494458:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655133440/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758481.494463:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758481.494465:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758481.494467:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655133440:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758481.494471:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655133568:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758481.494480:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758481.494486:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758481/real 1787758481] req@ffff8c05a9f78e00 x1874588655133568/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758481.494493:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655133568/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758481.494498:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758481.494500:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758481.494501:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655133568:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758481.494504:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655133696:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758481.494513:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758481.494519:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758481/real 1787758481] req@ffff8c05a9f7b800 x1874588655133696/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758481.494525:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655133696/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758481.494530:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758481.494532:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758481.494534:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655133696:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758481.494546:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655133824:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758481.494555:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758481.494561:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758481/real 1787758481] req@ffff8c05a9f79500 x1874588655133824/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758481.494568:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655133824/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758481.494573:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758481.494575:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758481.494577:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655133824:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758481.621966:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758482.070050:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758482.645975:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758483.094096:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758483.669984:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758484.118062:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758484.693961:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758485.142126:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758485.718013:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758486.166097:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758486.614008:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758486.614020:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758486.614025:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758486.614036:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11543 00000100:00080000:1.0:1787758486.614049:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758486.614057:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758486.614105:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655133952/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758486.614141:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758486.614144:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758486.614146:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758486.614149:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11543 00000100:00080000:1.0:1787758486.614152:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758486.614156:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758486.614165:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758486.614180:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588655134080/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758486.614188:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758486.614191:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758486.614192:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758486.614195:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11543 00000100:00080000:1.0:1787758486.614198:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758486.614202:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758486.614205:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758486.614217:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655134208/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758486.614221:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655133952:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758486.614231:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758486.614235:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758486.614237:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758486.614241:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11543 00000100:00080000:1.0:1787758486.614244:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758486.614247:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758486.614253:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758486.614262:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655134336/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758486.614268:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758486.614271:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758486.614272:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758486.614278:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11543 00000100:00080000:1.0:1787758486.614294:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758486.614297:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758486.614303:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787758486.614304:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758486.614328:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655134464/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758486.614338:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758486.614341:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758486.614343:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758486.614347:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11543 00000100:00080000:1.0:1787758486.614350:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787758486.614352:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758486/real 1787758486] req@ffff8c05b2d98380 x1874588655133952/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758486.614356:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758486.614367:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758486.614386:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655134592/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758486.614391:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655133952/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758486.614432:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758486.614433:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758486.614445:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787758486.614445:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787758486.614450:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655133952:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758486.614451:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787758486.614459:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655134080:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758486.614476:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758486.614484:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758486/real 1787758486] req@ffff8c05b2d9a300 x1874588655134080/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758486.614492:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588655134080/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758486.614498:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758486.614500:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758486.614503:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655134080:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758486.614544:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655134208:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758486.614557:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758486.614564:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758486/real 1787758486] req@ffff8c05b2d9aa00 x1874588655134208/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758486.614572:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655134208/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758486.614577:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758486.614580:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758486.614582:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655134208:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758486.614585:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655134336:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758486.614596:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758486.614602:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758486/real 1787758486] req@ffff8c05b2d9bb80 x1874588655134336/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758486.614609:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655134336/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758486.614614:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758486.614616:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758486.614618:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655134336:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758486.614622:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655134464:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758486.614632:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758486.614638:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758486/real 1787758486] req@ffff8c05b2d98700 x1874588655134464/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758486.614644:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655134464/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758486.614650:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758486.614651:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758486.614653:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655134464:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758486.614656:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655134592:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758486.614678:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758486.614685:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758486/real 1787758486] req@ffff8c05b2d9b100 x1874588655134592/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758486.614691:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655134592/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758486.614696:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758486.614698:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758486.614701:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655134592:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758486.615586:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11543 00000100:00080000:1.0:1787758486.615593:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758486.615599:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758486.615607:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758486.615630:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655134720/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758486.615651:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758486.615655:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758486.615656:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758486.615659:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11543 00000100:00080000:1.0:1787758486.615661:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758486.615664:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758486.615667:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758486.615674:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655134848/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758486.615703:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655134720:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758486.615723:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758486.615733:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758486/real 1787758486] req@ffff8c05b2d99880 x1874588655134720/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758486.615741:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655134720/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758486.615747:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758486.615749:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758486.615751:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655134720:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758486.615755:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655134848:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758486.615770:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758486.615779:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758486/real 1787758486] req@ffff8c05b2d99f80 x1874588655134848/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758486.615791:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655134848/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758486.615800:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758486.615804:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758486.615807:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655134848:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758486.741971:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758487.190199:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758487.766009:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758488.214167:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758488.789999:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758489.238064:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758489.813969:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758490.262081:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758490.837961:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758491.286081:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758491.734046:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758491.734056:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758491.734061:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758491.734077:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11548 00000100:00080000:3.0:1787758491.734090:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758491.734098:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758491.734138:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655134976/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758491.734167:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758491.734171:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758491.734174:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758491.734180:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11548 00000100:00080000:3.0:1787758491.734189:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758491.734194:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758491.734205:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758491.734220:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655134976:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787758491.734221:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655135104/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758491.734246:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758491.734249:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758491.734252:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787758491.734301:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758491.734340:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758491/real 1787758491] req@ffff8c05a9f7a300 x1874588655134976/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758491.734344:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11548 00000100:00100000:2.0:1787758491.734358:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655134976/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758491.734369:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758491.734373:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758491.734376:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655134976:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787758491.734396:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655135104:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758491.734412:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758491.734421:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758491/real 1787758491] req@ffff8c05a9f79880 x1874588655135104/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758491.734429:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655135104/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758491.734438:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758491.734440:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758491.734442:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655135104:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758491.734478:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758491.734482:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758491.734487:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758491.734533:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655135232/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758491.734550:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758491.734553:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758491.734555:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758491.734559:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11548 00000100:00080000:3.0:1787758491.734563:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758491.734567:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758491.734572:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758491.734584:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655135232:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758491.734585:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655135360/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758491.734611:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758491.734615:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758491.734616:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758491.734622:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11548 00000100:00080000:3.0:1787758491.734626:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758491.734633:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:2.0:1787758491.734636:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:3.0:1787758491.734640:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758491.734660:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758491/real 1787758491] req@ffff8c05a9f78700 x1874588655135232/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758491.734660:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655135488/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758491.734704:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758491.734723:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758491.734725:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758491.734737:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11548 00000100:00100000:2.0:1787758491.734738:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655135232/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758491.734749:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758491.734765:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:2.0:1787758491.734766:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758491.734776:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000008:00100000:3.0:1787758491.734777:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758491.734790:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655135232:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758491.734798:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655135616/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758491.734807:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655135360:192.168.201.107@tcp:8:kworker.0 00000100:00080000:3.0:1787758491.734808:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758491.734818:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758491.734821:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758491.734827:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11548 00000100:00080000:3.0:1787758491.734832:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787758491.734838:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758491.734843:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758491.734849:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758491.734852:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758491/real 1787758491] req@ffff8c05a9f7b480 x1874588655135360/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758491.734866:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655135744/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758491.734870:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655135360/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758491.734921:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758491.734938:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787758491.734950:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787758491.734951:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655135360:192.168.201.107@tcp:8:kworker.0 00000100:00080000:3.0:1787758491.734963:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758491.734967:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758491.734975:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11548 00000100:00100000:2.0:1787758491.734976:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655135488:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758491.734985:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758491.734992:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758491.734997:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787758491.735010:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758491.735012:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655135872/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758491.735047:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758491/real 1787758491] req@ffff8c05a9f78380 x1874588655135488/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758491.735065:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655135488/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758491.735071:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758491.735074:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758491.735077:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655135488:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758491.735088:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655135616:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758491.735106:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758491.735113:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758491/real 1787758491] req@ffff8c05a9f78a80 x1874588655135616/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758491.735121:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655135616/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758491.735126:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758491.735128:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758491.735130:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655135616:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758491.735134:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655135744:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758491.735144:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758491.735150:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758491/real 1787758491] req@ffff8c05a9f7a680 x1874588655135744/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758491.735156:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655135744/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758491.735162:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758491.735164:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758491.735165:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655135744:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758491.735169:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655135872:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758491.735179:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758491.735185:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758491/real 1787758491] req@ffff8c05a9f7aa00 x1874588655135872/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758491.735192:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655135872/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758491.735197:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758491.735199:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758491.735201:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655135872:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758491.861999:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758492.310101:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758492.885971:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758493.334170:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758493.910034:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758494.358126:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758494.933941:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758495.382179:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758495.958013:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758496.406095:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758496.853998:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758496.854009:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758496.854014:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758496.854023:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11553 00000100:00080000:3.0:1787758496.854034:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758496.854042:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758496.854086:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655136000/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758496.854118:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758496.854122:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758496.854124:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758496.854129:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11553 00000100:00080000:3.0:1787758496.854133:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758496.854138:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758496.854147:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758496.854165:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655136128/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758496.854167:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655136000:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758496.854184:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758496.854189:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758496.854191:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758496.854196:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11553 00000100:00080000:3.0:1787758496.854200:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758496.854205:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758496.854211:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758496.854224:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655136256/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758496.854237:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787758496.854239:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758496.854244:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758496.854247:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758496.854253:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11553 00000100:00080000:3.0:1787758496.854259:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758496.854264:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758496.854269:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758496.854281:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655136384/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758496.854290:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758496/real 1787758496] req@ffff8c05a9f7aa00 x1874588655136000/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758496.854297:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758496.854304:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758496.854307:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758496.854314:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11553 00000100:00080000:3.0:1787758496.854319:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758496.854324:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758496.854329:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758496.854340:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655136512/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758496.854348:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655136000/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758496.854355:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758496.854362:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758496.854364:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758496.854364:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758496.854370:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787758496.854374:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11553 00000100:00100000:2.0:1787758496.854376:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655136000:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758496.854380:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758496.854387:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758496.854392:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758496.854401:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655136128:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758496.854408:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655136640/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787758496.854424:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758496.854434:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787758496.854440:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758496/real 1787758496] req@ffff8c05a9f7a680 x1874588655136128/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758496.854441:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758496.854447:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787758496.854457:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655136128/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758496.854458:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11553 00000100:00080000:3.0:1787758496.854466:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758496.854470:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758496.854475:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:2.0:1787758496.854477:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758496.854484:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655136128:192.168.201.107@tcp:38:kworker.0 00000008:00100000:3.0:1787758496.854486:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758496.854490:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655136256:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758496.854505:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655136768/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758496.854519:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787758496.854521:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758496.854526:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758496.854529:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787758496.854534:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758496/real 1787758496] req@ffff8c05a9f78a80 x1874588655136256/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758496.854536:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11553 00000100:00080000:3.0:1787758496.854544:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787758496.854551:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655136256/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758496.854552:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758496.854562:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787758496.854563:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758496.854569:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758496.854572:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655136256:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758496.854578:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655136384:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758496.854583:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655136896/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787758496.854595:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758496.854606:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758496/real 1787758496] req@ffff8c05a9f78380 x1874588655136384/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758496.854615:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655136384/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758496.854625:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758496.854630:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758496.854633:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655136384:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758496.854640:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655136512:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758496.854661:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758496.854673:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758496/real 1787758496] req@ffff8c05a9f7b480 x1874588655136512/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758496.854684:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655136512/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758496.854690:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758496.854692:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758496.854694:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655136512:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758496.854707:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655136640:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758496.854722:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758496.854728:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758496/real 1787758496] req@ffff8c05a9f78700 x1874588655136640/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758496.854736:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655136640/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758496.854741:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758496.854744:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758496.854746:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655136640:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758496.854749:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655136768:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758496.854760:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758496.854765:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758496/real 1787758496] req@ffff8c05a9f79880 x1874588655136768/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758496.854773:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655136768/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758496.854777:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758496.854779:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758496.854781:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655136768:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758496.854784:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655136896:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758496.854793:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758496.854798:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758496/real 1787758496] req@ffff8c05a9f7a300 x1874588655136896/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758496.854806:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655136896/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758496.854811:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758496.854813:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758496.854814:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655136896:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758496.981972:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758497.430160:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758498.005968:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758498.454085:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758499.029995:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758499.478198:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758500.054001:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758500.502092:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758501.078036:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758501.526155:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758501.973992:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758501.974003:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758501.974008:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758501.974019:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11558 00000100:00080000:1.0:1787758501.974030:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758501.974037:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758501.974079:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655137024/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758501.974105:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758501.974108:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758501.974109:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758501.974112:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11558 00000100:00080000:1.0:1787758501.974115:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758501.974117:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758501.974125:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758501.974135:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655137152/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758501.974142:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758501.974145:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758501.974146:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758501.974149:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11558 00000100:00080000:1.0:1787758501.974151:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758501.974153:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758501.974156:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758501.974166:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655137280/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758501.974173:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758501.974176:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758501.974177:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758501.974180:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11558 00000100:00080000:1.0:1787758501.974183:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758501.974186:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758501.974190:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758501.974200:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655137408/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758501.974202:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655137024:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758501.974221:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758501.974223:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758501.974225:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758501.974228:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11558 00000100:00080000:1.0:1787758501.974233:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758501.974236:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758501.974241:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758501.974250:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655137536/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758501.974258:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758501.974261:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758501.974264:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758501.974267:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11558 00000100:00080000:1.0:1787758501.974269:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758501.974271:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758501.974275:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758501.974284:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655137664/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787758501.974294:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758501.974305:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758501.974309:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758501.974311:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758501.974316:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11558 00000100:00080000:1.0:1787758501.974325:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758501.974329:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758501.974335:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758501.974346:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655137792/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758501.974354:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758501/real 1787758501] req@ffff8c05b2d98000 x1874588655137024/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758501.974360:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758501.974379:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758501.974385:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758501.974398:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11558 00000100:00080000:1.0:1787758501.974402:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787758501.974405:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655137024/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758501.974409:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758501.974412:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758501.974421:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655137920/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758501.974438:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758501.974444:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758501.974449:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655137024:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787758501.974456:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655137152:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758501.974474:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758501.974485:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758501/real 1787758501] req@ffff8c05b2d9b480 x1874588655137152/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758501.974498:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655137152/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758501.974506:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758501.974510:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758501.974513:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655137152:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758501.974520:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655137280:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758501.974545:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758501.974556:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758501/real 1787758501] req@ffff8c05b2d99f80 x1874588655137280/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758501.974564:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655137280/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758501.974570:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758501.974572:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758501.974574:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655137280:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758501.974597:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655137408:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758501.974611:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758501.974617:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758501/real 1787758501] req@ffff8c05b2d99880 x1874588655137408/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758501.974625:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655137408/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758501.974630:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758501.974631:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758501.974633:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655137408:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758501.974637:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655137536:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758501.974647:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758501.974653:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758501/real 1787758501] req@ffff8c05b2d9b100 x1874588655137536/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758501.974665:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655137536/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758501.974670:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758501.974672:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758501.974674:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655137536:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758501.974677:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655137664:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758501.974687:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758501.974692:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758501/real 1787758501] req@ffff8c05b2d98700 x1874588655137664/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758501.974699:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655137664/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758501.974704:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758501.974706:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758501.974708:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655137664:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758501.974711:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655137792:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758501.974721:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758501.974726:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758501/real 1787758501] req@ffff8c05b2d9bb80 x1874588655137792/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758501.974733:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655137792/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758501.974743:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758501.974745:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758501.974747:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655137792:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758501.974750:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655137920:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758501.974760:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758501.974765:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758501/real 1787758501] req@ffff8c05b2d9aa00 x1874588655137920/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758501.974772:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655137920/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758501.974778:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758501.974780:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758501.974781:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655137920:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758502.101995:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758502.550033:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758503.126036:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758503.574082:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758504.149961:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758504.598040:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758505.174005:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758505.622189:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758506.197974:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758506.646141:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758507.093993:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758507.094002:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787758507.094006:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758507.094014:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11563 00000100:00080000:2.0:1787758507.094025:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758507.094032:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758507.094069:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588655138048/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758507.094097:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758507.094099:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758507.094101:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758507.094103:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11563 00000100:00080000:2.0:1787758507.094106:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758507.094108:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758507.094115:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758507.094124:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588655138176/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758507.094131:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758507.094133:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758507.094134:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758507.094137:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11563 00000100:00080000:2.0:1787758507.094138:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758507.094141:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758507.094143:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758507.094148:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588655138304/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758507.094155:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758507.094158:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758507.094159:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758507.094162:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11563 00000100:00080000:2.0:1787758507.094165:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758507.094168:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758507.094171:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758507.094177:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588655138432/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758507.094183:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758507.094185:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758507.094186:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758507.094188:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11563 00000100:00080000:2.0:1787758507.094191:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787758507.094195:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655138048:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787758507.094196:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758507.094202:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758507.094209:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588655138560/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758507.094214:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758507.094217:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758507.094218:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758507.094221:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11563 00000100:00080000:2.0:1787758507.094223:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758507.094225:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758507.094228:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758507.094237:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588655138688/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758507.094243:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758507.094245:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787758507.094247:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758507.094249:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11563 00000100:00080000:2.0:1787758507.094252:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758507.094255:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758507.094257:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787758507.094265:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758507.094266:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588655138816/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758507.094276:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758507.094279:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758507.094280:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758507.094284:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11563 00000100:00080000:2.0:1787758507.094287:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758507.094289:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758507.094292:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758507.094299:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588655138944/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758507.094313:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758507/real 1787758507] req@ffff8c05b2e70e00 x1874588655138048/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758507.094329:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588655138048/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758507.094338:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758507.094340:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758507.094344:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655138048:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787758507.094349:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655138176:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758507.094364:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758507.094375:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758507/real 1787758507] req@ffff8c05b2e70700 x1874588655138176/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758507.094388:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588655138176/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758507.094396:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758507.094400:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758507.094403:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655138176:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758507.094410:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655138304:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758507.094430:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758507.094439:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758507/real 1787758507] req@ffff8c05b2e73480 x1874588655138304/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758507.094447:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588655138304/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758507.094457:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758507.094459:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758507.094461:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655138304:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758507.094483:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655138432:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758507.094495:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758507.094501:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758507/real 1787758507] req@ffff8c05b2e71500 x1874588655138432/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758507.094508:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588655138432/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758507.094513:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758507.094515:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758507.094517:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655138432:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758507.094520:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655138560:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758507.094530:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758507.094536:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758507/real 1787758507] req@ffff8c05b2e73b80 x1874588655138560/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758507.094543:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588655138560/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758507.094547:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758507.094549:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758507.094551:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655138560:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758507.094555:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655138688:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758507.094564:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758507.094570:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758507/real 1787758507] req@ffff8c05b2e71c00 x1874588655138688/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758507.094580:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588655138688/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758507.094588:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758507.094591:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758507.094594:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655138688:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758507.094601:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655138816:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758507.094621:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758507.094632:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758507/real 1787758507] req@ffff8c05b2e71880 x1874588655138816/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758507.094643:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588655138816/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758507.094648:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758507.094650:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758507.094652:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655138816:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758507.094656:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655138944:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758507.094669:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758507.094676:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758507/real 1787758507] req@ffff8c05b2e70380 x1874588655138944/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758507.094683:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588655138944/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758507.094690:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758507.094693:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758507.094696:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655138944:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758507.222006:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758507.670164:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758508.245984:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758508.694090:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758509.270037:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758509.718157:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758510.293956:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758510.742058:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758511.318011:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758511.766136:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758512.214040:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758512.214055:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758512.214061:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758512.214076:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11568 00000100:00080000:0.0:1787758512.214091:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758512.214101:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758512.214163:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655139072/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758512.214203:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758512.214207:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758512.214210:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758512.214216:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11568 00000100:00080000:0.0:1787758512.214221:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758512.214228:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758512.214241:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758512.214263:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655139200/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758512.214267:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655139072:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758512.214294:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758512.214299:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758512.214301:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758512.214309:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11568 00000100:00080000:0.0:1787758512.214323:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758512.214332:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758512.214340:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787758512.214355:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758512.214364:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588655139328/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758512.214381:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758512.214386:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758512.214390:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758512.214395:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11568 00000100:00080000:0.0:1787758512.214402:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758512.214410:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758512.214417:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758512.214418:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758512/real 1787758512] req@ffff8c05a9f76680 x1874588655139072/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758512.214447:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588655139456/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758512.214447:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655139072/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758512.214502:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758512.214507:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787758512.214511:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787758512.214511:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655139072:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758512.214525:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758512.214529:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758512.214538:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11568 00000100:00100000:2.0:1787758512.214540:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655139200:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758512.214552:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758512.214563:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:2.0:1787758512.214571:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:0.0:1787758512.214577:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758512.214590:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758512/real 1787758512] req@ffff8c05a9f74a80 x1874588655139200/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758512.214605:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588655139584/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758512.214605:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655139200/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758512.214664:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758512.214687:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787758512.214689:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787758512.214693:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655139200:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758512.214699:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758512.214703:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787758512.214704:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655139328:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758512.214720:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11568 00000100:00080000:0.0:1787758512.214729:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758512.214734:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:2.0:1787758512.214739:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:0.0:1787758512.214741:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758512.214752:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758512/real 1787758512] req@ffff8c05a9f74e00 x1874588655139328/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758512.214762:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588655139712/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758512.214788:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588655139328/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758512.214801:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758512.214804:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758512.214807:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787758512.214811:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787758512.214814:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787758512.214816:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655139328:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758512.214825:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11568 00000100:00100000:2.0:1787758512.214828:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655139456:192.168.201.107@tcp:8:kworker.0 00000100:00080000:0.0:1787758512.214834:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758512.214842:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758512.214849:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787758512.214850:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758512.214860:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758512/real 1787758512] req@ffff8c05a9f77100 x1874588655139456/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758512.214871:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588655139840/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758512.214872:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588655139456/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758512.214913:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758512.214916:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758512.214921:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655139456:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758512.214946:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655139584:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758512.214977:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787758512.214984:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758512.214988:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758512.214993:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758512.215002:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11568 00000100:00100000:2.0:1787758512.215003:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758512/real 1787758512] req@ffff8c05a9f76d80 x1874588655139584/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758512.215013:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787758512.215022:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588655139584/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758512.215024:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758512.215039:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787758512.215049:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758512.215054:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758512.215057:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655139584:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758512.215067:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655139712:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758512.215068:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655139968/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787758512.215096:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758512.215105:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758512/real 1787758512] req@ffff8c05a9f75180 x1874588655139712/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758512.215113:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588655139712/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758512.215120:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758512.215123:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758512.215126:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655139712:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758512.215143:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655139840:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758512.215154:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758512.215161:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758512/real 1787758512] req@ffff8c05a9f74000 x1874588655139840/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758512.215168:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588655139840/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758512.215173:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758512.215175:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758512.215178:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655139840:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758512.215182:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655139968:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758512.215191:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758512.215197:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758512/real 1787758512] req@ffff8c05a9f75880 x1874588655139968/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758512.215204:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655139968/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758512.215212:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758512.215215:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758512.215219:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655139968:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758512.341971:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758512.790045:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758513.365988:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758513.814132:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758514.389969:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758514.838041:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758515.413996:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758515.862159:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758516.437991:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758516.886096:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758517.333980:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758517.333988:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758517.333991:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758517.333999:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11573 00000100:00080000:0.0:1787758517.334010:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758517.334016:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758517.334051:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588655140096/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758517.334077:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758517.334079:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758517.334081:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758517.334084:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11573 00000100:00080000:0.0:1787758517.334090:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758517.334094:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758517.334102:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758517.334113:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655140224/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758517.334120:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758517.334123:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758517.334125:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758517.334128:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11573 00000100:00080000:0.0:1787758517.334130:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758517.334133:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758517.334135:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758517.334141:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588655140352/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758517.334148:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758517.334150:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758517.334154:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787758517.334156:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655140096:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758517.334157:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11573 00000100:00080000:0.0:1787758517.334162:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758517.334168:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758517.334171:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758517.334178:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588655140480/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758517.334185:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758517.334187:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758517.334189:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758517.334191:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11573 00000100:00080000:0.0:1787758517.334194:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758517.334196:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758517.334199:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758517.334206:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588655140608/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758517.334211:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758517.334213:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758517.334214:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758517.334217:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11573 00000100:00080000:0.0:1787758517.334223:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758517.334225:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758517.334228:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758517.334235:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588655140736/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787758517.334244:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758517.334250:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758517.334253:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758517.334255:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758517.334258:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11573 00000100:00080000:0.0:1787758517.334261:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758517.334264:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758517.334268:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758517.334274:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588655140864/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758517.334281:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758517.334283:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758517.334285:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758517.334288:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11573 00000100:00080000:0.0:1787758517.334290:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758517.334292:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758517.334295:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758517.334298:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758517/real 1787758517] req@ffff8c05a9f75500 x1874588655140096/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758517.334305:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655140992/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758517.334336:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588655140096/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758517.334351:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758517.334355:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758517.334361:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655140096:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787758517.334370:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655140224:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758517.334394:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758517.334408:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758517/real 1787758517] req@ffff8c05a9f75880 x1874588655140224/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758517.334421:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655140224/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758517.334431:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758517.334435:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758517.334438:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655140224:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758517.334462:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655140352:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758517.334484:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758517.334496:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758517/real 1787758517] req@ffff8c05a9f74000 x1874588655140352/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758517.334510:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588655140352/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758517.334518:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758517.334521:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758517.334525:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655140352:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758517.334532:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655140480:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758517.334551:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758517.334564:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758517/real 1787758517] req@ffff8c05a9f75180 x1874588655140480/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758517.334577:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588655140480/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758517.334604:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758517.334607:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758517.334611:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655140480:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758517.334617:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655140608:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758517.334640:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758517.334653:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758517/real 1787758517] req@ffff8c05a9f76d80 x1874588655140608/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758517.334666:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588655140608/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758517.334675:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758517.334678:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758517.334682:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655140608:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758517.334688:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655140736:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758517.334707:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758517.334719:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758517/real 1787758517] req@ffff8c05a9f77100 x1874588655140736/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758517.334732:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588655140736/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758517.334742:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758517.334745:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758517.334749:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655140736:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758517.334754:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655140864:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758517.334773:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758517.334783:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758517/real 1787758517] req@ffff8c05a9f74e00 x1874588655140864/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758517.334797:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588655140864/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758517.334806:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758517.334810:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758517.334814:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655140864:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758517.334820:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655140992:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758517.334838:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758517.334849:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758517/real 1787758517] req@ffff8c05a9f74a80 x1874588655140992/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758517.334861:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655140992/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758517.334869:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758517.334873:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758517.334905:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655140992:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758517.461991:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758517.910090:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758518.486013:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758518.934093:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758519.509964:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758519.958122:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758520.533999:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758520.982138:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758521.557970:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758522.006099:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758522.454067:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758522.454082:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758522.454089:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758522.454102:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11578 00000100:00080000:1.0:1787758522.454117:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758522.454126:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758522.454177:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588655141120/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758522.454219:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758522.454224:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758522.454227:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758522.454232:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11578 00000100:00080000:1.0:1787758522.454238:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758522.454243:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758522.454254:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758522.454275:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655141120:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758522.454291:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655141248/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758522.454310:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758522.454314:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758522.454317:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758522.454323:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11578 00000100:00080000:1.0:1787758522.454329:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758522.454334:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758522.454341:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758522.454356:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655141376/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787758522.454367:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758522.454380:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758522.454385:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758522.454388:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758522.454396:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11578 00000100:00080000:1.0:1787758522.454402:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758522.454408:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758522.454414:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758522.454416:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758522/real 1787758522] req@ffff8c05b2d9a300 x1874588655141120/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758522.454437:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655141504/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758522.454439:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588655141120/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758522.454519:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758522.454524:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758522.454525:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758522.454528:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758522.454538:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787758522.454539:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11578 00000100:00080000:1.0:1787758522.454548:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787758522.454552:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655141120:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758522.454560:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758522.454568:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758522.454584:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655141632/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758522.454588:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655141248:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758522.454614:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758522.454619:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758522.454623:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787758522.454631:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758522.454637:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11578 00000100:00080000:1.0:1787758522.454644:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787758522.454646:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758522/real 1787758522] req@ffff8c05b2d98380 x1874588655141248/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758522.454654:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758522.454671:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655141248/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787758522.454671:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787758522.454690:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758522.454694:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758522.454704:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655141248:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758522.454704:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655141760/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758522.454723:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655141376:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758522.454752:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787758522.454758:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758522.454763:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758522.454768:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787758522.454774:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758522/real 1787758522] req@ffff8c05b2d9aa00 x1874588655141376/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758522.454778:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11578 00000100:00080000:1.0:1787758522.454790:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787758522.454798:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655141376/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758522.454803:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787758522.454823:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787758522.454823:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787758522.454836:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758522.454840:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655141376:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758522.454847:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655141504:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758522.454853:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655141888/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787758522.454865:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758522.454867:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758522.454873:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787758522.454902:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758522/real 1787758522] req@ffff8c05b2d9bb80 x1874588655141504/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758522.454922:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655141504/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758522.454929:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758522.454942:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11578 00000100:00080000:0.0:1787758522.454943:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758522.454948:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758522.454952:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758522.454960:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655141504:192.168.201.107@tcp:8:kworker.0 00000100:00080000:1.0:1787758522.454961:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758522.454980:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758522.454994:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655141632:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758522.455002:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655142016/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787758522.455019:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758522.455040:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758522/real 1787758522] req@ffff8c05b2d98700 x1874588655141632/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758522.455049:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655141632/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758522.455055:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758522.455058:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758522.455060:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655141632:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758522.455064:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655141760:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758522.455075:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758522.455082:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758522/real 1787758522] req@ffff8c05b2d9b100 x1874588655141760/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758522.455089:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655141760/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758522.455098:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758522.455100:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758522.455102:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655141760:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758522.455106:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655141888:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758522.455123:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758522.455130:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758522/real 1787758522] req@ffff8c05b2d99880 x1874588655141888/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758522.455137:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655141888/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758522.455146:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758522.455148:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758522.455150:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655141888:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758522.455160:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655142016:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758522.455172:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758522.455178:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758522/real 1787758522] req@ffff8c05b2d99f80 x1874588655142016/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758522.455186:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655142016/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758522.455191:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758522.455193:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758522.455196:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655142016:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758522.582025:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758523.030011:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758523.605925:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758524.054098:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758524.630007:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758525.078117:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758525.653996:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758526.102176:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758526.678013:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758527.126089:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758527.574005:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758527.574017:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758527.574023:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758527.574034:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11583 00000100:00080000:3.0:1787758527.574046:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758527.574055:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758527.574096:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655142144/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758527.574132:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758527.574136:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758527.574139:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758527.574144:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11583 00000100:00080000:3.0:1787758527.574148:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758527.574154:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758527.574165:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758527.574173:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655142144:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787758527.574185:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655142272/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758527.574201:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758527.574205:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758527.574208:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758527.574213:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11583 00000100:00080000:3.0:1787758527.574218:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758527.574223:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758527.574229:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758527.574244:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655142400/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758527.574257:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758527.574261:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758527.574263:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758527.574269:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11583 00000100:00080000:3.0:1787758527.574273:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758527.574278:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758527.574284:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787758527.574352:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758527.574367:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655142528/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758527.574382:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758527.574387:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758527.574389:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758527.574394:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11583 00000100:00080000:3.0:1787758527.574566:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758527.574573:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758527.574580:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758527.574596:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655142656/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758527.574607:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758527.574651:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758527.574655:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758527.574661:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11583 00000100:00080000:3.0:1787758527.574667:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758527.574672:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758527.574678:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758527.574691:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655142784/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758527.574702:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758527.574705:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758527.574707:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758527.574711:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11583 00000100:00080000:3.0:1787758527.574714:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758527.574718:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758527.574723:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758527.574735:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655142912/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758527.574744:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758527.574748:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758527.574750:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758527.574754:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11583 00000100:00080000:3.0:1787758527.574757:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758527.574760:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758527.574764:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758527.574775:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655143040/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758527.575498:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758527/real 1787758527] req@ffff8c05a9f79500 x1874588655142144/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758527.575518:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655142144/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758527.575527:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758527.575531:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758527.575534:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655142144:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758527.575562:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655142272:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758527.575600:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758527.575610:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758527/real 1787758527] req@ffff8c05a9f7b800 x1874588655142272/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758527.575618:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655142272/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758527.575624:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758527.575626:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758527.575629:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655142272:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758527.575633:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655142400:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758527.575646:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758527.575652:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758527/real 1787758527] req@ffff8c05a9f78e00 x1874588655142400/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758527.575665:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655142400/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758527.575670:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758527.575672:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758527.575673:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655142400:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758527.575678:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655142528:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758527.575691:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758527.575698:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758527/real 1787758527] req@ffff8c05a9f7ad80 x1874588655142528/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758527.575705:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655142528/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758527.575714:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758527.575718:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758527.575722:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655142528:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758527.575728:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655142656:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758527.575749:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758527.575761:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758527/real 1787758527] req@ffff8c05a9f7b100 x1874588655142656/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758527.575774:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655142656/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758527.575785:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758527.575788:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758527.575790:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655142656:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758527.575795:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655142784:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758527.575810:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758527.575817:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758527/real 1787758527] req@ffff8c05a9f79f80 x1874588655142784/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758527.575825:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655142784/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758527.575830:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758527.575832:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758527.575835:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655142784:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758527.575838:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655142912:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758527.575848:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758527.575855:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758527/real 1787758527] req@ffff8c05a9f7bb80 x1874588655142912/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758527.575862:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655142912/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758527.575867:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758527.575869:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758527.575871:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655142912:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758527.575891:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655143040:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758527.575910:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758527.575919:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758527/real 1787758527] req@ffff8c05a9f79180 x1874588655143040/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758527.575926:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655143040/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758527.575932:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758527.575934:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758527.575936:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655143040:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758527.701980:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758528.150105:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758528.726015:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758529.174120:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758529.750004:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758530.198017:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758530.773990:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758531.222087:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758531.798008:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758532.246118:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758532.693973:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758532.693985:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758532.693990:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758532.694013:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11589 00000100:00080000:1.0:1787758532.694022:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758532.694037:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758532.694084:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655143168/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758532.694108:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758532.694111:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758532.694113:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758532.694116:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11589 00000100:00080000:1.0:1787758532.694120:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758532.694125:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758532.694137:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758532.694147:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655143168:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758532.694156:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655143296/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758532.694170:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758532.694174:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758532.694176:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758532.694179:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11589 00000100:00080000:1.0:1787758532.694183:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758532.694186:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758532.694191:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758532.694212:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655143424/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758532.694220:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758532.694223:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758532.694224:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758532.694227:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11589 00000100:00080000:1.0:1787758532.694230:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758532.694232:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758532.694236:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758532.694250:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655143552/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758532.694257:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655143296:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758532.694271:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758532.694276:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758532.694278:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758532.694285:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11589 00000100:00080000:1.0:1787758532.694289:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787758532.694290:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655143424:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758532.694296:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758532.694303:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758532.694333:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655143552:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758532.694334:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655143680/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758532.694346:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758532.694348:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758532.694350:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758532.694354:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11589 00000100:00080000:1.0:1787758532.694358:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758532.694362:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758532.694366:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655143680:192.168.201.107@tcp:38:kworker.0 00000008:00100000:1.0:1787758532.694368:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758532.694384:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655143808/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758532.694394:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758532.694396:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758532.694398:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758532.694402:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11589 00000100:00080000:1.0:1787758532.694405:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787758532.694407:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655143808:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758532.694412:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758532.694417:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758532.694429:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655143936/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758532.694442:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758532.694445:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758532.694447:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787758532.694452:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655143936:192.168.201.107@tcp:8:kworker.0 00000100:00080000:1.0:1787758532.694452:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11589 00000100:00080000:1.0:1787758532.694456:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758532.694460:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758532.694464:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758532.694476:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655144064/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758532.694507:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655144064:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758532.821990:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758533.270125:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758533.846011:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758534.294168:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758534.870003:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758535.318093:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787758535.766200:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.107:988 00000400:00000100:3.0:1787758535.766270:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.107@tcp at host 192.168.201.107 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787758535.766303:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:3.0:1787758535.766338:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:3.0:1787758535.766362:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:3.0:1787758535.766372:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:3.0:1787758535.766381:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:3.0:1787758535.766390:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:3.0:1787758535.766405:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:3.0:1787758535.766431:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:0.0:1787758535.766441:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758532/real 1787758535] req@ffff8c05b2d98700 x1874588655143808/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758535.766474:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655143808/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758535.766489:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758535.766495:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758535.766500:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655143808:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758535.766510:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758532/real 1787758535] req@ffff8c05b2d9b100 x1874588655143680/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758535.766522:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655143680/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758535.766529:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758535.766532:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758535.766535:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655143680:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758535.766539:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758532/real 1787758535] req@ffff8c05b2d99880 x1874588655143552/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758535.766548:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655143552/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758535.766556:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758535.766560:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758535.766563:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655143552:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758535.766571:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758532/real 1787758535] req@ffff8c05b2d98000 x1874588655143296/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758535.766584:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655143296/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758535.766592:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758535.766596:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758535.766599:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655143296:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758535.766606:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758532/real 1787758535] req@ffff8c05b2d99f80 x1874588655143424/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758535.766618:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655143424/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758535.766716:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758535.766729:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758535.766736:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655143424:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758535.766751:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758532/real 1787758535] req@ffff8c05b2d9b480 x1874588655143168/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758535.766780:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655143168/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758535.766788:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758535.766792:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758535.766795:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655143168:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787758535.766842:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758532/real 1787758535] req@ffff8c05b2d9aa00 x1874588655144064/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758535.766857:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655144064/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758535.766866:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758535.766871:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758535.766874:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655144064:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758535.766940:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758532/real 1787758535] req@ffff8c05b2d9bb80 x1874588655143936/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758535.766960:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655143936/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758535.766969:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758535.766972:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758535.766975:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655143936:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758535.893969:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758536.342181:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758536.917979:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758537.366125:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758537.814064:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758537.814079:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758537.814084:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758537.814106:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11594 00000100:00080000:0.0:1787758537.814119:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758537.814127:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758537.814168:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655144192/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758537.814208:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758537.814214:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1787758537.814317:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655144192:192.168.201.107@tcp:250:kworker.0 00000800:00000100:1.0:1787758537.814420:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758537.814488:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758537/real 1787758537] req@ffff8c05a9f76680 x1874588655144192/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758537.814489:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758537.814507:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11594 00000100:00080000:0.0:1787758537.814518:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758537.814526:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758537.814532:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655144192/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787758537.814545:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787758537.814570:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758537.814578:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758537.814586:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655144320/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758537.814590:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655144192:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758537.814635:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758537.814641:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787758537.814643:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655144320:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758537.814646:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758537.814662:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11594 00000100:00080000:0.0:1787758537.814670:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758537.814678:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758537.814687:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787758537.814700:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758537.814713:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655144448/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758537.814727:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758537/real 1787758537] req@ffff8c05a9f77800 x1874588655144320/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758537.814738:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758537.814753:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758537.814759:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758537.814769:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11594 00000100:00100000:1.0:1787758537.814772:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655144320/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758537.814778:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758537.814792:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:1.0:1787758537.814798:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787758537.814804:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787758537.814811:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758537.814821:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655144320:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758537.814830:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655144576/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758537.814850:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758537.814852:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655144448:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758537.814861:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758537.814868:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787758537.814935:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758537.814954:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11594 00000100:00100000:1.0:1787758537.814959:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758537/real 1787758537] req@ffff8c05a9f74380 x1874588655144448/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758537.814966:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758537.814982:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758537.814990:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655144448/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787758537.814992:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787758537.815014:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787758537.815023:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588655144704/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758537.815026:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758537.815039:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655144448:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758537.815054:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758537.815062:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758537.815065:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758537.815072:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655144576:192.168.201.107@tcp:8:kworker.0 00000100:00080000:0.0:1787758537.815078:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11594 00000100:00080000:0.0:1787758537.815089:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758537.815097:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758537.815106:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787758537.815122:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758537.815131:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588655144832/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758537.815147:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758537/real 1787758537] req@ffff8c05a9f76300 x1874588655144576/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758537.815154:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758537.815160:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758537.815163:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758537.815178:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11594 00000100:00080000:0.0:1787758537.815189:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758537.815196:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655144576/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758537.815199:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758537.815213:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787758537.815221:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758537.815259:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758537.815267:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588655144960/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758537.815270:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655144576:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758537.815294:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655144704:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758537.815324:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758537.815330:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758537.815333:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758537.815342:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11594 00000800:00000100:1.0:1787758537.815350:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758537.815353:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758537.815363:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758537.815371:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758537.815380:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758537/real 1787758537] req@ffff8c05a9f77b80 x1874588655144704/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758537.815396:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655145088/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758537.815410:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588655144704/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758537.815427:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758537.815432:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758537.815436:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655144704:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758537.815456:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655144832:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758537.815482:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758537.815493:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758537/real 1787758537] req@ffff8c05a9f76a00 x1874588655144832/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758537.815505:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588655144832/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758537.815514:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758537.815517:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758537.815520:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655144832:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758537.815525:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655144960:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758537.815545:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758537.815561:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758537/real 1787758537] req@ffff8c05a9f75c00 x1874588655144960/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758537.815578:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588655144960/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758537.815591:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758537.815596:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758537.815601:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655144960:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758537.815623:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655145088:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758537.815657:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758537.815676:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758537/real 1787758537] req@ffff8c05a9f74380 x1874588655145088/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758537.815696:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655145088/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758537.815718:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758537.815722:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758537.815727:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655145088:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758537.942036:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758538.390137:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758538.965974:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758539.414053:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758539.989994:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758540.438164:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758541.013952:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758541.462041:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758542.037989:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758542.486211:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758542.934054:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758542.934064:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758542.934068:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758542.934077:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11599 00000100:00080000:3.0:1787758542.934088:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758542.934095:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758542.934158:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655145216/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758542.934184:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758542.934187:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758542.934188:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758542.934192:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11599 00000100:00080000:3.0:1787758542.934195:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758542.934197:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758542.934207:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758542.934218:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655145344/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758542.934226:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758542.934230:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758542.934231:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758542.934234:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11599 00000100:00080000:3.0:1787758542.934236:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758542.934240:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758542.934243:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655145216:192.168.201.107@tcp:250:kworker.0 00000008:00100000:3.0:1787758542.934244:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758542.934254:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655145472/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758542.934262:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758542.934265:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758542.934267:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758542.934270:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11599 00000100:00080000:3.0:1787758542.934273:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758542.934275:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758542.934278:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758542.934285:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655145600/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758542.934302:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758542.934304:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758542.934305:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758542.934308:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11599 00000100:00080000:3.0:1787758542.934312:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758542.934315:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758542.934319:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787758542.934327:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758542.934331:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655145728/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758542.934339:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758542.934343:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758542.934345:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758542.934347:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11599 00000100:00080000:3.0:1787758542.934351:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758542.934354:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758542.934358:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758542.934366:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655145856/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758542.934373:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758542.934379:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758542/real 1787758542] req@ffff8c05a9f79180 x1874588655145216/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758542.934379:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758542.934382:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758542.934386:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11599 00000100:00080000:3.0:1787758542.934388:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758542.934390:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758542.934393:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758542.934406:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655145984/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758542.934419:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758542.934421:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758542.934422:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758542.934426:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655145216/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758542.934430:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11599 00000100:00080000:3.0:1787758542.934436:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758542.934443:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:1.0:1787758542.934448:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787758542.934448:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787758542.934454:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758542.934458:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655146112/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758542.934462:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655145216:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758542.934477:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655145344:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758542.934500:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758542.934512:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758542/real 1787758542] req@ffff8c05a9f7bb80 x1874588655145344/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758542.934521:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655145344/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758542.934531:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758542.934534:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758542.934537:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655145344:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758542.934557:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655145472:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758542.934575:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758542.934587:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758542/real 1787758542] req@ffff8c05a9f79f80 x1874588655145472/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758542.934601:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655145472/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758542.934611:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758542.934614:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758542.934618:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655145472:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758542.934624:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655145600:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758542.934641:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758542.934649:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758542/real 1787758542] req@ffff8c05a9f7b100 x1874588655145600/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758542.934657:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655145600/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758542.934663:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758542.934666:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758542.934668:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655145600:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758542.934672:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655145728:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758542.934684:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758542.934690:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758542/real 1787758542] req@ffff8c05a9f7ad80 x1874588655145728/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758542.934698:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655145728/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758542.934703:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758542.934705:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758542.934707:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655145728:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758542.934711:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655145856:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758542.934721:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758542.934727:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758542/real 1787758542] req@ffff8c05a9f78e00 x1874588655145856/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758542.934735:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655145856/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758542.934740:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758542.934742:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758542.934744:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655145856:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758542.934747:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655145984:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758542.934756:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758542.934762:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758542/real 1787758542] req@ffff8c05a9f7b800 x1874588655145984/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758542.934770:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655145984/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758542.934776:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758542.934778:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758542.934779:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655145984:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758542.934783:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655146112:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758542.934792:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758542.934798:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758542/real 1787758542] req@ffff8c05a9f79500 x1874588655146112/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758542.934805:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655146112/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758542.934813:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758542.934816:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758542.934820:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655146112:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758543.061939:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758543.510043:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758544.085974:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758544.534201:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758545.109968:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758545.558101:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758546.133999:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758546.582107:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758547.157970:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758547.606116:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758548.053945:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758548.053956:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758548.053960:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758548.053971:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11604 00000100:00080000:3.0:1787758548.053982:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758548.053989:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758548.054033:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655146240/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758548.054077:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758548.054081:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758548.054083:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758548.054088:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11604 00000100:00080000:3.0:1787758548.054091:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758548.054095:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758548.054103:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758548.054115:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655146368/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758548.054124:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758548.054126:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758548.054128:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758548.054131:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11604 00000100:00080000:3.0:1787758548.054134:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758548.054136:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758548.054139:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758548.054149:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655146496/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758548.054150:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655146240:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758548.054165:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758548.054168:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758548.054169:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758548.054171:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11604 00000100:00080000:3.0:1787758548.054173:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758548.054176:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758548.054179:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758548.054186:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655146624/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758548.054192:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758548.054194:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758548.054195:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758548.054198:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11604 00000100:00080000:3.0:1787758548.054201:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758548.054204:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758548.054207:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758548.054215:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655146752/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758548.054222:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758548.054224:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758548.054227:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787758548.054229:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758548.054233:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11604 00000100:00080000:3.0:1787758548.054236:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758548.054239:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758548.054242:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758548.054251:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655146880/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758548.054257:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758548.054259:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758548.054260:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758548.054263:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11604 00000100:00080000:3.0:1787758548.054265:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758548.054269:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758548.054272:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758548.054279:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758548/real 1787758548] req@ffff8c05a9f7a300 x1874588655146240/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758548.054283:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655147008/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758548.054306:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758548.054308:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758548.054309:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758548.054313:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655146240/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758548.054314:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11604 00000100:00080000:3.0:1787758548.054320:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758548.054325:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758548.054331:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787758548.054336:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758548.054342:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758548.054342:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655147136/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758548.054351:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655146240:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758548.054361:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655146368:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758548.054380:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758548.054388:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758548/real 1787758548] req@ffff8c05a9f79880 x1874588655146368/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758548.054397:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655146368/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758548.054402:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758548.054404:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758548.054406:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655146368:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758548.054426:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655146496:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758548.054437:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758548.054443:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758548/real 1787758548] req@ffff8c05a9f78700 x1874588655146496/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758548.054450:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655146496/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758548.054456:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758548.054458:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758548.054460:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655146496:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758548.054465:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655146624:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758548.054476:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758548.054482:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758548/real 1787758548] req@ffff8c05a9f7b480 x1874588655146624/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758548.054489:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655146624/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758548.054495:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758548.054496:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758548.054498:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655146624:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758548.054501:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655146752:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758548.054511:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758548.054517:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758548/real 1787758548] req@ffff8c05a9f78380 x1874588655146752/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758548.054523:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655146752/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758548.054528:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758548.054530:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758548.054532:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655146752:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758548.054535:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655146880:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758548.054545:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758548.054550:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758548/real 1787758548] req@ffff8c05a9f78a80 x1874588655146880/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758548.054559:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655146880/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758548.054564:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758548.054565:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758548.054567:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655146880:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758548.054570:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655147008:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758548.054580:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758548.054586:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758548/real 1787758548] req@ffff8c05a9f7a680 x1874588655147008/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758548.054592:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655147008/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758548.054597:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758548.054599:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758548.054601:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655147008:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758548.054605:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655147136:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758548.054615:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758548.054620:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758548/real 1787758548] req@ffff8c05a9f7aa00 x1874588655147136/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758548.054628:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655147136/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758548.054638:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758548.054640:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758548.054642:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655147136:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758548.182006:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758548.630155:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758549.205995:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758549.654100:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758550.229987:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758550.678173:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758551.253955:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758551.702096:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758552.277983:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758552.726111:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758553.173995:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758553.174009:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758553.174015:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758553.174029:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11609 00000100:00080000:3.0:1787758553.174042:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758553.174052:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758553.174102:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655147264/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758553.174130:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758553.174133:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758553.174135:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758553.174140:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11609 00000100:00080000:3.0:1787758553.174145:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758553.174149:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758553.174161:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758553.174176:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655147392/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758553.174182:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655147264:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758553.174187:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758553.174190:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758553.174192:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758553.174196:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11609 00000100:00080000:3.0:1787758553.174199:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758553.174203:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758553.174208:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758553.174219:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655147520/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758553.174228:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758553.174231:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758553.174233:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758553.174237:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11609 00000100:00080000:3.0:1787758553.174239:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758553.174243:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758553.174247:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787758553.174257:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758553.174257:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655147648/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758553.174266:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758553.174269:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758553.174271:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758553.174274:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11609 00000100:00080000:3.0:1787758553.174277:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758553.174281:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758553.174285:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758553.174295:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655147776/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758553.174303:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758553.174307:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758553/real 1787758553] req@ffff8c05a9f7aa00 x1874588655147264/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758553.174307:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758553.174309:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758553.174313:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11609 00000100:00080000:3.0:1787758553.174317:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758553.174320:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758553.174324:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758553.174327:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655147264/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758553.174334:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655147904/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758553.174339:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758553.174342:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758553.174343:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787758553.174346:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787758553.174348:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655147264:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758553.174348:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758553.174352:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11609 00000100:00080000:3.0:1787758553.174355:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758553.174358:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758553.174362:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758553.174375:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655147392:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758553.174382:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655148032/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758553.174391:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758553.174395:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758553.174397:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787758553.174399:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758553.174401:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11609 00000100:00080000:3.0:1787758553.174404:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758553.174409:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758553.174412:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758553/real 1787758553] req@ffff8c05a9f7a680 x1874588655147392/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787758553.174414:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758553.174425:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655147392/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758553.174425:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655148160/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758553.174442:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758553.174445:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758553.174448:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655147392:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758553.174454:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655147520:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758553.174475:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758553.174487:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758553/real 1787758553] req@ffff8c05a9f78a80 x1874588655147520/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758553.174496:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655147520/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758553.174502:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758553.174504:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758553.174507:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655147520:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758553.174511:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655147648:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758553.174526:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758553.174532:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758553/real 1787758553] req@ffff8c05a9f78380 x1874588655147648/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758553.174540:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655147648/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758553.174545:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758553.174547:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758553.174549:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655147648:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758553.174552:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655147776:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758553.174562:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758553.174571:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758553/real 1787758553] req@ffff8c05a9f7b480 x1874588655147776/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758553.174583:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655147776/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758553.174591:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758553.174594:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758553.174598:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655147776:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758553.174604:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655147904:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758553.174625:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758553.174634:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758553/real 1787758553] req@ffff8c05a9f78700 x1874588655147904/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758553.174643:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655147904/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758553.174648:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758553.174651:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758553.174653:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655147904:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758553.174666:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655148032:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758553.174678:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758553.174684:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758553/real 1787758553] req@ffff8c05a9f79880 x1874588655148032/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758553.174692:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655148032/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758553.174698:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758553.174700:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758553.174702:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655148032:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758553.174705:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655148160:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758553.174714:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758553.174719:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758553/real 1787758553] req@ffff8c05a9f7a300 x1874588655148160/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758553.174726:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655148160/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758553.174732:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758553.174733:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758553.174735:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655148160:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758553.301977:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758553.750081:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758554.325998:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758554.774120:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758555.350011:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758555.798130:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758556.373964:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758556.822111:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758557.398004:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758557.846077:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758558.294015:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758558.294026:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758558.294031:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758558.294041:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11614 00000100:00080000:1.0:1787758558.294053:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758558.294060:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758558.294107:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655148288/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758558.294141:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758558.294145:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758558.294146:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758558.294149:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11614 00000100:00080000:1.0:1787758558.294153:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758558.294156:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758558.294166:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758558.294178:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588655148416/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758558.294187:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758558.294190:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758558.294191:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758558.294196:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11614 00000100:00100000:0.0:1787758558.294197:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655148288:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758558.294200:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758558.294207:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758558.294211:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758558.294220:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655148544/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758558.294228:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758558.294231:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758558.294233:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758558.294235:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11614 00000100:00080000:1.0:1787758558.294238:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758558.294240:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758558.294244:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758558.294251:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655148672/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758558.294257:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758558.294260:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758558.294261:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758558.294263:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11614 00000100:00080000:1.0:1787758558.294266:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758558.294270:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:0.0:1787758558.294275:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:1.0:1787758558.294277:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758558.294286:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655148800/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758558.294292:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758558.294295:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758558.294297:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758558.294301:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11614 00000100:00080000:1.0:1787758558.294315:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758558.294318:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758558.294323:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758558.294332:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758558/real 1787758558] req@ffff8c05b2d98380 x1874588655148288/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758558.294334:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655148928/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758558.294365:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758558.294368:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758558.294369:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758558.294374:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11614 00000100:00100000:0.0:1787758558.294375:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655148288/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758558.294376:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758558.294383:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758558.294394:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787758558.294395:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758558.294402:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758558.294406:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655148288:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758558.294413:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655149056/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758558.294425:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758558.294428:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758558.294430:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787758558.294435:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655148416:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758558.294438:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11614 00000100:00080000:1.0:1787758558.294445:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758558.294450:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758558.294457:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787758558.294459:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758558.294471:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758558/real 1787758558] req@ffff8c05b2d9a300 x1874588655148416/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758558.294478:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655149184/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758558.294485:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588655148416/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758558.294501:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758558.294505:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758558.294509:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655148416:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758558.294514:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655148544:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758558.294529:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758558.294536:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758558/real 1787758558] req@ffff8c05b2d9bb80 x1874588655148544/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758558.294544:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655148544/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758558.294549:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758558.294552:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758558.294554:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655148544:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758558.294557:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655148672:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758558.294568:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758558.294574:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758558/real 1787758558] req@ffff8c05b2d9aa00 x1874588655148672/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758558.294582:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655148672/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758558.294587:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758558.294589:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758558.294591:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655148672:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758558.294594:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655148800:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758558.294605:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758558.294611:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758558/real 1787758558] req@ffff8c05b2d9b480 x1874588655148800/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758558.294618:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655148800/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758558.294624:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758558.294625:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758558.294633:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655148800:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758558.294637:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655148928:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758558.294648:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758558.294655:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758558/real 1787758558] req@ffff8c05b2d99f80 x1874588655148928/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758558.294662:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655148928/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758558.294672:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758558.294674:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758558.294681:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655148928:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758558.294685:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655149056:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758558.294696:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758558.294703:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758558/real 1787758558] req@ffff8c05b2d98000 x1874588655149056/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758558.294710:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655149056/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758558.294716:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758558.294717:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758558.294719:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655149056:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758558.294734:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655149184:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758558.294744:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758558.294751:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758558/real 1787758558] req@ffff8c05b2d99880 x1874588655149184/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758558.294758:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655149184/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758558.294763:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758558.294765:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758558.294767:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655149184:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758558.421967:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758558.870088:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758559.445997:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758559.894026:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758560.469966:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758560.918101:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758561.493946:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758561.942055:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758562.517953:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758562.966099:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758563.414078:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758563.414088:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758563.414095:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758563.414110:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11619 00000100:00080000:3.0:1787758563.414124:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758563.414133:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758563.414176:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655149312/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758563.414208:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758563.414212:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758563.414214:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758563.414219:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11619 00000100:00080000:3.0:1787758563.414223:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758563.414228:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758563.414239:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758563.414255:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655149440/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758563.414268:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758563.414273:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758563.414275:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758563.414280:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11619 00000100:00080000:3.0:1787758563.414284:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787758563.414289:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655149312:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758563.414291:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758563.414301:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758563.414316:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655149568/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787758563.414389:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758563.414440:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758563/real 1787758563] req@ffff8c05a9f79500 x1874588655149312/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758563.414463:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655149312/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758563.414476:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758563.414482:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758563.414488:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655149312:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787758563.414497:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655149440:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758563.414524:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787758563.414530:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758563.414532:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787758563.414563:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758563/real 1787758563] req@ffff8c05a9f7b800 x1874588655149440/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758563.414575:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787758563.414582:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655149440/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758563.414585:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11619 00000100:00080000:3.0:1787758563.414595:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758563.414601:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758563.414603:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787758563.414612:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000008:00100000:3.0:1787758563.414614:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758563.414620:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655149440:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758563.414635:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655149696/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758563.414648:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758563.414652:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758563.414655:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787758563.414656:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655149568:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758563.414663:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11619 00000100:00080000:3.0:1787758563.414670:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758563.414674:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758563.414681:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787758563.414693:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758563.414696:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655149824/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758563.414712:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758563/real 1787758563] req@ffff8c05a9f78e00 x1874588655149568/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758563.414728:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758563.414733:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787758563.414737:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655149568/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758563.414737:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758563.414748:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11619 00000100:00080000:3.0:1787758563.414755:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758563.414757:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758563.414762:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787758563.414768:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000008:00100000:3.0:1787758563.414785:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758563.414791:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655149568:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758563.414803:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655149952/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758563.414805:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655149696:192.168.201.107@tcp:8:kworker.0 00000100:00080000:3.0:1787758563.414823:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758563.414827:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758563.414829:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758563.414835:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11619 00000100:00080000:3.0:1787758563.414839:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787758563.414843:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758563.414848:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758563.414856:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758563.414864:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758563/real 1787758563] req@ffff8c05a9f7ad80 x1874588655149696/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758563.414870:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655150080/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758563.414911:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758563.414915:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758563.414916:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758563.414921:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11619 00000100:00080000:3.0:1787758563.414928:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758563.414935:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758563.414941:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655149696/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787758563.414942:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787758563.414958:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787758563.414961:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655150208/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758563.414969:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758563.414979:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655149696:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758563.415002:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655149824:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758563.415029:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758563.415047:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758563/real 1787758563] req@ffff8c05a9f7b100 x1874588655149824/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758563.415071:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655149824/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758563.415082:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758563.415086:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758563.415090:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655149824:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758563.415098:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655149952:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758563.415123:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758563.415141:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758563/real 1787758563] req@ffff8c05a9f79f80 x1874588655149952/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758563.415155:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655149952/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758563.415164:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758563.415168:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758563.415171:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655149952:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758563.415178:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655150080:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758563.415200:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758563.415212:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758563/real 1787758563] req@ffff8c05a9f7bb80 x1874588655150080/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758563.415225:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655150080/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758563.415241:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758563.415245:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758563.415249:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655150080:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758563.415256:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655150208:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758563.415277:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758563.415289:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758563/real 1787758563] req@ffff8c05a9f79180 x1874588655150208/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758563.415303:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655150208/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758563.415312:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758563.415315:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758563.415319:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655150208:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758563.542020:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758563.990080:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758564.565943:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758565.014068:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758565.589933:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758566.038065:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758566.613959:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758567.062084:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758567.637986:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758568.086038:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758568.533963:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758568.533975:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758568.533980:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758568.533992:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11624 00000100:00080000:3.0:1787758568.534002:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758568.534010:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758568.534051:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655150336/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758568.534083:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758568.534086:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758568.534088:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758568.534092:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11624 00000100:00080000:3.0:1787758568.534095:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758568.534098:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758568.534107:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758568.534118:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655150336:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787758568.534119:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655150464/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758568.534132:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758568.534135:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758568.534136:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758568.534139:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11624 00000100:00080000:3.0:1787758568.534143:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758568.534146:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758568.534150:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758568.534159:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655150592/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758568.534167:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758568.534170:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758568.534171:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758568.534173:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11624 00000100:00080000:3.0:1787758568.534175:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758568.534178:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758568.534182:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758568.534189:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655150720/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787758568.534192:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758568.534201:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758568.534204:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758568.534206:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758568.534210:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11624 00000100:00080000:3.0:1787758568.534213:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758568.534216:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758568.534219:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758568.534226:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655150848/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758568.534238:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758568.534240:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758568.534242:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787758568.534247:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758568/real 1787758568] req@ffff8c05a9f79180 x1874588655150336/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758568.534248:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11624 00000100:00080000:3.0:1787758568.534252:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758568.534258:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758568.534264:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758568.534275:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655150976/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758568.534282:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787758568.534284:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655150336/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758568.534285:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758568.534288:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758568.534295:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11624 00000100:00080000:3.0:1787758568.534301:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758568.534307:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758568.534311:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787758568.534312:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787758568.534320:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655151104/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758568.534321:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758568.534333:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655150336:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758568.534334:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758568.534341:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758568.534350:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758568.534355:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11624 00000100:00080000:3.0:1787758568.534358:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758568.534361:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758568.534364:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758568.534370:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655150464:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758568.534375:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655151232/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787758568.534398:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758568.534411:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758568/real 1787758568] req@ffff8c05a9f7bb80 x1874588655150464/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758568.534421:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655150464/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758568.534428:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758568.534430:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758568.534433:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655150464:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758568.534437:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655150592:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758568.534449:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758568.534455:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758568/real 1787758568] req@ffff8c05a9f79f80 x1874588655150592/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758568.534463:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655150592/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758568.534468:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758568.534470:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758568.534472:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655150592:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758568.534476:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655150720:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758568.534487:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758568.534494:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758568/real 1787758568] req@ffff8c05a9f7b100 x1874588655150720/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758568.534501:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655150720/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758568.534507:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758568.534508:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758568.534510:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655150720:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758568.534514:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655150848:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758568.534524:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758568.534530:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758568/real 1787758568] req@ffff8c05a9f7ad80 x1874588655150848/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758568.534538:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655150848/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758568.534543:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758568.534544:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758568.534546:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655150848:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758568.534549:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655150976:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758568.534559:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758568.534564:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758568/real 1787758568] req@ffff8c05a9f78e00 x1874588655150976/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758568.534571:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655150976/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758568.534578:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758568.534580:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758568.534582:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655150976:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758568.534585:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655151104:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758568.534603:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758568.534610:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758568/real 1787758568] req@ffff8c05a9f7b800 x1874588655151104/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758568.534617:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655151104/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758568.534622:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758568.534625:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758568.534629:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655151104:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758568.534656:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655151232:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758568.534678:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758568.534702:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758568/real 1787758568] req@ffff8c05a9f79500 x1874588655151232/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758568.534715:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655151232/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758568.534724:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758568.534727:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758568.534732:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655151232:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758568.661973:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758569.110103:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758569.686034:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758570.134097:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758570.709950:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758571.158137:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758571.734006:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758572.182097:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758572.757970:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758573.206184:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758573.654030:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758573.654041:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758573.654045:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758573.654057:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11630 00000100:00080000:0.0:1787758573.654067:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758573.654075:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758573.654116:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655151360/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758573.654143:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758573.654145:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758573.654147:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758573.654150:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11630 00000100:00080000:0.0:1787758573.654153:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758573.654155:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758573.654164:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758573.654175:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655151488/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758573.654183:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758573.654188:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758573.654189:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758573.654189:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655151360:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758573.654197:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11630 00000100:00080000:0.0:1787758573.654200:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758573.654204:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758573.654207:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758573.654216:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655151616/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758573.654223:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758573.654226:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758573.654228:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758573.654230:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11630 00000100:00080000:0.0:1787758573.654251:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758573.654256:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:1.0:1787758573.654262:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:0.0:1787758573.654263:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758573.654291:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588655151744/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758573.654301:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758573.654303:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758573.654305:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758573.654309:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11630 00000100:00080000:0.0:1787758573.654313:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758573.654316:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758573.654321:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758573/real 1787758573] req@ffff8c05a9f77800 x1874588655151360/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787758573.654322:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758573.654337:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588655151872/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758573.654344:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655151360/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758573.654348:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758573.654356:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758573.654359:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758573.654364:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758573.654365:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11630 00000100:00080000:0.0:1787758573.654370:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758573.654374:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787758573.654377:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758573.654382:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655151360:192.168.201.107@tcp:250:kworker.0 00000008:00100000:0.0:1787758573.654383:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758573.654396:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588655152000/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758573.654402:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758573.654404:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758573.654406:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758573.654410:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655151488:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758573.654413:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11630 00000100:00080000:0.0:1787758573.654417:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758573.654420:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758573.654424:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758573.654433:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588655152128/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787758573.654437:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758573.654447:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758573.654450:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758573/real 1787758573] req@ffff8c05a9f76680 x1874588655151488/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758573.654452:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758573.654454:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758573.654461:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11630 00000100:00080000:0.0:1787758573.654468:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758573.654473:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655151488/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758573.654477:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:1.0:1787758573.654487:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787758573.654488:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787758573.654492:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758573.654496:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655151488:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758573.654508:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655151616:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758573.654509:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588655152256/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787758573.654530:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758573.654541:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758573/real 1787758573] req@ffff8c05a9f74a80 x1874588655151616/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758573.654552:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655151616/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758573.654558:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758573.654561:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758573.654563:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655151616:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758573.654566:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655151744:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758573.654578:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758573.654589:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758573/real 1787758573] req@ffff8c05a9f74e00 x1874588655151744/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758573.654613:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588655151744/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758573.654619:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758573.654621:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758573.654623:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655151744:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758573.654628:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655151872:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758573.654642:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758573.654649:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758573/real 1787758573] req@ffff8c05a9f77100 x1874588655151872/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758573.654657:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588655151872/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758573.654663:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758573.654664:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758573.654666:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655151872:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758573.654669:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655152000:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758573.654679:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758573.654685:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758573/real 1787758573] req@ffff8c05a9f76d80 x1874588655152000/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758573.654693:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588655152000/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758573.654698:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758573.654700:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758573.654702:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655152000:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758573.654715:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655152128:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758573.654736:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758573.654743:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758573/real 1787758573] req@ffff8c05a9f75180 x1874588655152128/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758573.654751:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588655152128/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758573.654761:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758573.654763:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758573.654765:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655152128:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758573.654769:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655152256:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758573.654779:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758573.654785:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758573/real 1787758573] req@ffff8c05a9f74000 x1874588655152256/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758573.654792:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588655152256/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758573.654797:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758573.654799:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758573.654800:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655152256:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758573.781995:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758574.230089:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758574.805996:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758575.254187:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758575.830008:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758576.278061:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758576.853995:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758577.302166:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758577.877996:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758578.326061:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758578.774042:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758578.774055:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758578.774060:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758578.774073:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11635 00000100:00080000:1.0:1787758578.774087:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758578.774095:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758578.774140:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655152384/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758578.774181:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758578.774185:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758578.774194:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758578.774198:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11635 00000100:00080000:1.0:1787758578.774202:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758578.774207:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758578.774219:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758578.774233:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655152512/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758578.774245:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655152384:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758578.774248:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758578.774257:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758578.774259:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758578.774264:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11635 00000100:00080000:1.0:1787758578.774267:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758578.774271:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758578.774276:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758578.774290:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655152640/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758578.774301:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758578.774305:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758578.774308:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758578.774312:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11635 00000100:00080000:1.0:1787758578.774331:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787758578.774339:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758578.774339:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758578.774348:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758578.774362:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655152768/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758578.774373:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758578.774377:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758578.774379:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758578.774383:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11635 00000100:00080000:1.0:1787758578.774388:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758578.774393:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758578.774399:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758578.774402:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758578/real 1787758578] req@ffff8c05b2d9b100 x1874588655152384/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758578.774414:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655152896/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758578.774428:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758578.774431:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758578.774433:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758578.774438:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11635 00000100:00100000:0.0:1787758578.774444:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655152384/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758578.774444:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758578.774452:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758578.774460:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787758578.774468:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787758578.774475:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655153024/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758578.774478:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758578.774488:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655152384:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758578.774517:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758578.774521:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758578.774522:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758578.774527:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11635 00000100:00080000:1.0:1787758578.774530:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758578.774535:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758578.774539:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655152512:192.168.201.107@tcp:38:kworker.0 00000008:00100000:1.0:1787758578.774543:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758578.774560:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655153152/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758578.774572:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787758578.774578:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758578.774580:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758578.774584:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758578.774590:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11635 00000100:00080000:1.0:1787758578.774594:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758578.774602:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758578.774603:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758578/real 1787758578] req@ffff8c05b2d98700 x1874588655152512/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787758578.774609:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758578.774628:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655152512/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758578.774634:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655153280/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758578.774650:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758578.774658:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758578.774664:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655152512:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758578.774672:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655152640:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758578.774698:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758578.774712:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758578/real 1787758578] req@ffff8c05b2d99880 x1874588655152640/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758578.774726:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655152640/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758578.774734:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758578.774738:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758578.774741:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655152640:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758578.774747:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655152768:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758578.774781:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758578.774795:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758578/real 1787758578] req@ffff8c05b2d98000 x1874588655152768/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758578.774809:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655152768/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758578.774818:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758578.774822:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758578.774825:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655152768:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758578.774832:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655152896:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758578.774853:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758578.774865:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758578/real 1787758578] req@ffff8c05b2d99f80 x1874588655152896/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758578.774920:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655152896/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758578.774931:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758578.774934:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758578.774938:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655152896:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758578.774944:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655153024:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758578.774967:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758578.774981:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758578/real 1787758578] req@ffff8c05b2d9b480 x1874588655153024/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758578.774993:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655153024/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758578.775002:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758578.775006:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758578.775009:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655153024:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758578.775030:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655153152:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758578.775052:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758578.775065:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758578/real 1787758578] req@ffff8c05b2d9aa00 x1874588655153152/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758578.775077:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655153152/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758578.775094:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758578.775098:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758578.775101:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655153152:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758578.775108:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655153280:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758578.775125:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758578.775133:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758578/real 1787758578] req@ffff8c05b2d9bb80 x1874588655153280/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758578.775145:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655153280/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758578.775154:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758578.775157:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758578.775161:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655153280:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758578.901959:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758579.350148:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758579.925969:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758580.374027:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758580.949940:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758581.398151:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758581.973947:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758582.422086:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758582.997963:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758583.446096:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758583.894014:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758583.894025:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758583.894029:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758583.894039:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11640 00000100:00080000:3.0:1787758583.894051:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758583.894058:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758583.894101:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655153408/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758583.894127:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758583.894130:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758583.894132:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758583.894134:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11640 00000100:00080000:3.0:1787758583.894137:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758583.894139:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758583.894147:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758583.894155:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655153536/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758583.894162:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758583.894165:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758583.894166:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758583.894168:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11640 00000100:00080000:3.0:1787758583.894170:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758583.894172:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758583.894174:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758583.894180:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655153664/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758583.894188:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758583.894191:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758583.894193:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758583.894195:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11640 00000100:00080000:3.0:1787758583.894198:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758583.894200:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758583.894203:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758583.894212:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655153792/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758583.894214:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655153408:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758583.894223:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758583.894226:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758583.894227:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758583.894230:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11640 00000100:00080000:3.0:1787758583.894231:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758583.894234:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758583.894236:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758583.894246:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655153920/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758583.894252:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758583.894254:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758583.894255:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758583.894258:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11640 00000100:00080000:3.0:1787758583.894260:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758583.894262:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758583.894266:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758583.894273:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655154048/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758583.894279:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758583.894282:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758583.894284:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758583.894286:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11640 00000100:00080000:3.0:1787758583.894288:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758583.894290:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758583.894292:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787758583.894299:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758583.894301:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655154176/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758583.894308:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758583.894312:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758583.894314:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758583.894318:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11640 00000100:00080000:3.0:1787758583.894323:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758583.894328:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758583.894333:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758583.894346:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655154304/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758583.894361:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758583/real 1787758583] req@ffff8c05a9f7a300 x1874588655153408/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758583.894386:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655153408/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758583.894398:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758583.894403:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758583.894409:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655153408:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787758583.894435:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655153536:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758583.894462:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758583.894478:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758583/real 1787758583] req@ffff8c05a9f79880 x1874588655153536/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758583.894492:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655153536/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758583.894501:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758583.894505:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758583.894508:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655153536:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758583.894515:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655153664:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758583.894537:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758583.894549:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758583/real 1787758583] req@ffff8c05a9f78700 x1874588655153664/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758583.894562:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655153664/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758583.894571:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758583.894575:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758583.894579:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655153664:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758583.894610:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655153792:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758583.894633:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758583.894647:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758583/real 1787758583] req@ffff8c05a9f7b480 x1874588655153792/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758583.894660:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655153792/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758583.894670:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758583.894673:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758583.894677:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655153792:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758583.894683:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655153920:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758583.894704:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758583.894717:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758583/real 1787758583] req@ffff8c05a9f78380 x1874588655153920/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758583.894730:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655153920/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758583.894739:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758583.894743:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758583.894746:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655153920:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758583.894753:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655154048:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758583.894773:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758583.894785:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758583/real 1787758583] req@ffff8c05a9f78a80 x1874588655154048/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758583.894797:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655154048/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758583.894807:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758583.894811:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758583.894814:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655154048:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758583.894821:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655154176:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758583.894843:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758583.894855:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758583/real 1787758583] req@ffff8c05a9f7a680 x1874588655154176/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758583.894868:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655154176/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758583.894904:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758583.894908:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758583.894912:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655154176:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758583.894920:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655154304:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758583.894943:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758583.894956:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758583/real 1787758583] req@ffff8c05a9f7aa00 x1874588655154304/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758583.894969:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655154304/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758583.894979:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758583.894982:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758583.894987:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655154304:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758584.021950:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758584.470082:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758585.045977:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758585.494067:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758586.069967:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758586.518080:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758587.093980:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758587.542142:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758588.117967:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758588.566067:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758589.013934:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758589.013945:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758589.013950:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758589.013960:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11645 00000100:00080000:3.0:1787758589.013973:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758589.013981:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758589.014034:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655154432/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758589.014060:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758589.014063:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758589.014064:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758589.014068:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11645 00000100:00080000:3.0:1787758589.014071:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758589.014074:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758589.014083:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758589.014093:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655154560/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758589.014102:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758589.014105:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758589.014107:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787758589.014112:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655154432:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758589.014113:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11645 00000100:00080000:3.0:1787758589.014120:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758589.014124:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758589.014128:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758589.014144:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655154688/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758589.014152:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758589.014155:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758589.014156:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758589.014159:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11645 00000100:00080000:3.0:1787758589.014162:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758589.014165:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758589.014168:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758589.014176:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655154816/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758589.014183:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758589.014185:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758589.014187:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758589.014189:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11645 00000100:00080000:3.0:1787758589.014191:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758589.014196:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:0.0:1787758589.014198:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:3.0:1787758589.014199:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758589.014209:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655154944/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758589.014217:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758589.014220:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758589.014221:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758589.014224:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11645 00000100:00080000:3.0:1787758589.014227:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758589.014229:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758589.014232:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758589.014241:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655155072/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758589.014247:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787758589.014252:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758589/real 1787758589] req@ffff8c05a9f7aa00 x1874588655154432/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758589.014252:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758589.014253:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758589.014261:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11645 00000100:00080000:3.0:1787758589.014265:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758589.014276:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758589.014281:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758589.014286:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655154432/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758589.014293:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655155200/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758589.014312:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758589.014312:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758589.014320:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758589.014333:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787758589.014334:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787758589.014341:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655154432:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758589.014342:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11645 00000100:00080000:3.0:1787758589.014350:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758589.014354:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758589.014357:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758589.014376:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655155328/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758589.014385:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655154560:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758589.014409:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758589.014420:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758589/real 1787758589] req@ffff8c05a9f7a680 x1874588655154560/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758589.014428:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655154560/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758589.014435:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758589.014437:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758589.014440:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655154560:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758589.014444:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655154688:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758589.014455:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758589.014462:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758589/real 1787758589] req@ffff8c05a9f78a80 x1874588655154688/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758589.014470:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655154688/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758589.014475:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758589.014476:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758589.014478:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655154688:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758589.014482:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655154816:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758589.014493:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758589.014499:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758589/real 1787758589] req@ffff8c05a9f78380 x1874588655154816/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758589.014506:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655154816/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758589.014512:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758589.014513:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758589.014515:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655154816:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758589.014518:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655154944:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758589.014528:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758589.014534:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758589/real 1787758589] req@ffff8c05a9f7b480 x1874588655154944/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758589.014546:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655154944/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758589.014551:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758589.014553:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758589.014555:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655154944:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758589.014558:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655155072:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758589.014567:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758589.014583:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758589/real 1787758589] req@ffff8c05a9f78700 x1874588655155072/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758589.014590:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655155072/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758589.014595:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758589.014597:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758589.014598:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655155072:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758589.014601:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655155200:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758589.014611:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758589.014616:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758589/real 1787758589] req@ffff8c05a9f79880 x1874588655155200/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758589.014622:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655155200/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758589.014632:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758589.014634:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758589.014636:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655155200:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758589.014649:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655155328:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758589.014660:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758589.014666:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758589/real 1787758589] req@ffff8c05a9f7a300 x1874588655155328/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758589.014673:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655155328/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758589.014678:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758589.014680:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758589.014682:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655155328:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758589.142039:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758589.590107:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758590.165966:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758590.614030:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758591.190029:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758591.638115:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758592.214001:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758592.662058:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758593.237985:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758593.686137:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758594.133984:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758594.133998:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758594.134003:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758594.134016:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11650 00000100:00080000:1.0:1787758594.134030:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758594.134039:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758594.134095:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588655155456/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758594.134121:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758594.134124:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758594.134127:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758594.134131:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11650 00000100:00080000:1.0:1787758594.134136:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758594.134141:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758594.134161:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758594.134176:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655155456:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758594.134181:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655155584/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758594.134192:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758594.134196:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758594.134197:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758594.134200:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11650 00000100:00080000:1.0:1787758594.134204:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758594.134206:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758594.134210:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758594.134221:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655155712/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758594.134228:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758594.134231:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758594.134244:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758594.134248:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11650 00000100:00080000:1.0:1787758594.134252:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758594.134255:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758594.134259:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787758594.134266:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758594.134270:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655155840/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758594.134278:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758594.134280:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758594.134282:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758594.134284:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11650 00000100:00080000:1.0:1787758594.134287:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758594.134290:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758594.134294:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758594.134300:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655155968/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758594.134312:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758594.134315:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758594.134316:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787758594.134321:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758594/real 1787758594] req@ffff8c05b2d9a300 x1874588655155456/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758594.134324:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11650 00000100:00080000:1.0:1787758594.134330:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758594.134337:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758594.134344:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758594.134355:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588655155456/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758594.134356:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655156096/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758594.134384:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758594.134387:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758594.134389:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758594.134390:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758594.134397:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11650 00000100:00080000:0.0:1787758594.134401:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787758594.134401:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758594.134408:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758594.134412:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655155456:192.168.201.107@tcp:250:kworker.0 00000008:00100000:1.0:1787758594.134413:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758594.134426:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655156224/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758594.134433:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758594.134435:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758594.134436:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758594.134439:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11650 00000100:00080000:1.0:1787758594.134441:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787758594.134445:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655155584:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758594.134446:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758594.134452:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758594.134462:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655156352/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787758594.134477:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758594.134492:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758594/real 1787758594] req@ffff8c05b2d98380 x1874588655155584/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758594.134506:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655155584/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758594.134515:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758594.134518:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758594.134522:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655155584:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758594.134528:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655155712:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758594.134544:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758594.134553:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758594/real 1787758594] req@ffff8c05b2d9bb80 x1874588655155712/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758594.134564:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655155712/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758594.134572:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758594.134575:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758594.134578:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655155712:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758594.134584:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655155840:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758594.134599:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758594.134608:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758594/real 1787758594] req@ffff8c05b2d9aa00 x1874588655155840/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758594.134620:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655155840/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758594.134628:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758594.134631:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758594.134635:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655155840:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758594.134641:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655155968:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758594.134657:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758594.134667:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758594/real 1787758594] req@ffff8c05b2d9b480 x1874588655155968/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758594.134680:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655155968/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758594.134688:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758594.134692:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758594.134695:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655155968:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758594.134700:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655156096:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758594.134715:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758594.134723:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758594/real 1787758594] req@ffff8c05b2d99f80 x1874588655156096/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758594.134736:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655156096/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758594.134745:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758594.134749:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758594.134752:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655156096:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758594.134758:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655156224:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758594.134774:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758594.134784:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758594/real 1787758594] req@ffff8c05b2d98000 x1874588655156224/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758594.134795:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655156224/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758594.134804:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758594.134807:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758594.134810:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655156224:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758594.134830:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655156352:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758594.134847:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758594.134856:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758594/real 1787758594] req@ffff8c05b2d99880 x1874588655156352/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758594.134868:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655156352/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758594.134901:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758594.134904:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758594.134908:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655156352:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758594.262007:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758594.710084:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758595.285973:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758595.734132:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758596.309970:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758596.758095:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758597.333977:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758597.782062:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758598.357970:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758598.806084:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758599.254034:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758599.254045:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758599.254051:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758599.254063:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11655 00000100:00080000:1.0:1787758599.254074:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758599.254081:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758599.254122:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655156480/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758599.254162:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758599.254166:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758599.254167:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758599.254171:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11655 00000100:00080000:1.0:1787758599.254174:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758599.254178:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758599.254188:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758599.254202:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655156608/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758599.254211:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787758599.254214:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655156480:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758599.254214:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758599.254217:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758599.254225:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11655 00000100:00080000:1.0:1787758599.254228:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758599.254232:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758599.254237:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758599.254254:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655156736/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758599.254275:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758599.254278:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758599.254280:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758599.254283:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11655 00000100:00080000:1.0:1787758599.254293:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758599.254296:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758599.254300:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758599.254308:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655156864/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758599.254315:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758599.254318:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758599.254320:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758599.254322:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11655 00000100:00080000:1.0:1787758599.254328:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758599.254334:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758599.254338:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758599.254350:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655156992/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758599.254361:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758599.254364:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758599.254365:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787758599.254369:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655156608:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758599.254371:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11655 00000100:00080000:1.0:1787758599.254377:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758599.254380:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758599.254384:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758599.254394:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655157120/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758599.254402:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758599.254406:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758599.254408:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758599.254413:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11655 00000100:00080000:1.0:1787758599.254419:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787758599.254424:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655156736:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758599.254427:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758599.254436:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758599.254456:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655157248/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758599.254470:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758599.254474:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758599.254477:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758599.254487:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11655 00000100:00100000:0.0:1787758599.254492:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655156864:192.168.201.107@tcp:8:kworker.0 00000100:00080000:1.0:1787758599.254492:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758599.254499:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758599.254504:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758599.254514:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655157376/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758599.254533:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655156992:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758599.254564:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655157120:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758599.254593:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655157248:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758599.254629:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655157376:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758599.381951:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758599.830145:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758600.405972:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758600.854201:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758601.429966:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758601.878304:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787758602.326203:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.107:988 00000400:00000100:2.0:1787758602.326251:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.107@tcp at host 192.168.201.107 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787758602.326267:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787758602.326311:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787758602.326327:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787758602.326339:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787758602.326351:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787758602.326362:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787758602.326373:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:0.0:1787758602.326385:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758599/real 1787758602] req@ffff8c05b2d9b100 x1874588655156608/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787758602.326385:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:0.0:1787758602.326420:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655156608/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758602.326435:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758602.326440:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758602.326452:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655156608:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758602.326462:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758599/real 1787758602] req@ffff8c05b2d99880 x1874588655156736/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758602.326476:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655156736/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758602.326483:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758602.326487:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758602.326490:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655156736:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758602.326497:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758599/real 1787758602] req@ffff8c05b2d98000 x1874588655156864/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758602.326509:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655156864/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758602.326517:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758602.326521:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758602.326524:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655156864:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758602.326531:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758599/real 1787758602] req@ffff8c05b2d99f80 x1874588655156992/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758602.326545:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655156992/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758602.326553:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758602.326557:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758602.326561:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655156992:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758602.326568:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758599/real 1787758602] req@ffff8c05b2d98700 x1874588655156480/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758602.326584:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655156480/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758602.326592:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758602.326595:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758602.326599:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655156480:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787758602.326636:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758599/real 1787758602] req@ffff8c05b2d9b480 x1874588655157120/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758602.326650:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655157120/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758602.326658:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758602.326662:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758602.326667:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655157120:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758602.326674:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758599/real 1787758602] req@ffff8c05b2d9aa00 x1874588655157248/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758602.326687:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655157248/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758602.326695:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758602.326699:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758602.326703:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655157248:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758602.326711:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758599/real 1787758602] req@ffff8c05b2d9bb80 x1874588655157376/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758602.326724:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655157376/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758602.326731:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758602.326735:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758602.326739:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655157376:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758602.454017:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758602.902105:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758603.477962:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758603.926106:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758604.374099:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758604.374110:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787758604.374114:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758604.374125:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11660 00000100:00080000:2.0:1787758604.374137:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758604.374144:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758604.374176:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73800 x1874588655157504/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758604.374202:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758604.374205:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758604.374206:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758604.374211:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11660 00000100:00080000:2.0:1787758604.374215:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758604.374220:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758604.374230:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758604.374248:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588655157632/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758604.374261:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787758604.374266:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655157504:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787758604.374278:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758604.374285:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758604.374292:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11660 00000100:00080000:2.0:1787758604.374298:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758604.374305:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758604.374310:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758604.374323:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588655157760/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758604.374339:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758604.374342:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758604.374343:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758604.374347:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11660 00000100:00080000:2.0:1787758604.374350:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758604.374353:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758604.374359:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758604.374368:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588655157888/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787758604.374380:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787758604.374384:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758604.374390:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758604.374393:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758604.374403:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11660 00000100:00080000:2.0:1787758604.374411:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758604.374416:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758604.374422:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758604.374435:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588655158016/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758604.374446:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758604.374450:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787758604.374456:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758604/real 1787758604] req@ffff8c05b2e73800 x1874588655157504/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758604.374457:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758604.374474:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11660 00000100:00080000:2.0:1787758604.374487:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758604.374493:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758604.374499:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758604.374501:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73800 x1874588655157504/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758604.374520:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588655158144/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758604.374522:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758604.374536:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758604.374549:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655157504:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787758604.374552:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758604.374562:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787758604.374564:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758604.374570:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11660 00000100:00080000:2.0:1787758604.374574:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758604.374578:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758604.374585:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758604.374593:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655157632:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758604.374602:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588655158272/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758604.374618:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758604.374621:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758604.374624:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758604.374629:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11660 00000800:00000100:0.0:1787758604.374636:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787758604.374638:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758604.374647:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758604.374655:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758604.374662:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758604/real 1787758604] req@ffff8c05b2e70380 x1874588655157632/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758604.374676:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588655158400/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758604.374682:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588655157632/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758604.374751:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758604.374760:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758604.374765:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655157632:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758604.374775:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655157760:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758604.374805:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758604.374817:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758604/real 1787758604] req@ffff8c05b2e71880 x1874588655157760/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758604.374829:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588655157760/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758604.374839:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758604.374843:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758604.374846:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655157760:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758604.374854:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655157888:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758604.374922:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758604.374935:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758604/real 1787758604] req@ffff8c05b2e71c00 x1874588655157888/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758604.374949:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588655157888/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758604.374959:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758604.374963:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758604.374967:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655157888:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758604.374975:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655158016:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758604.374997:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758604.375007:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758604/real 1787758604] req@ffff8c05b2e73b80 x1874588655158016/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758604.375019:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588655158016/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758604.375027:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758604.375031:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758604.375035:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655158016:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758604.375041:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655158144:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758604.375082:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758604.375096:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758604/real 1787758604] req@ffff8c05b2e71500 x1874588655158144/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758604.375110:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588655158144/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758604.375119:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758604.375123:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758604.375126:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655158144:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758604.375149:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655158272:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758604.375171:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758604.375185:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758604/real 1787758604] req@ffff8c05b2e73480 x1874588655158272/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758604.375197:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588655158272/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758604.375206:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758604.375210:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758604.375213:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655158272:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758604.375220:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655158400:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758604.375239:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758604.375250:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758604/real 1787758604] req@ffff8c05b2e70700 x1874588655158400/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758604.375262:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588655158400/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758604.375270:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758604.375274:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758604.375277:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655158400:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758604.501994:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758604.950105:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758605.525948:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758605.974143:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758606.549959:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758606.998084:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758607.573944:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758608.022101:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758608.597977:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758609.046121:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758609.493985:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758609.493994:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787758609.493997:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758609.494006:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11665 00000100:00080000:2.0:1787758609.494016:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758609.494023:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758609.494058:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588655158528/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758609.494083:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758609.494086:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758609.494087:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758609.494090:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11665 00000100:00080000:2.0:1787758609.494092:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758609.494095:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758609.494103:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758609.494112:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588655158656/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758609.494121:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758609.494124:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758609.494126:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758609.494131:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11665 00000100:00080000:2.0:1787758609.494137:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787758609.494140:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655158528:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787758609.494141:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758609.494151:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758609.494168:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588655158784/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758609.494176:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758609.494178:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758609.494179:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758609.494184:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11665 00000100:00080000:2.0:1787758609.494186:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758609.494188:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758609.494191:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758609.494198:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588655158912/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758609.494204:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758609.494206:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758609.494208:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758609.494210:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11665 00000100:00080000:2.0:1787758609.494217:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758609.494220:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758609.494223:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758609.494229:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588655159040/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758609.494235:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758609.494237:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758609.494239:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758609.494242:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11665 00000100:00080000:2.0:1787758609.494245:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787758609.494252:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787758609.494253:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758609.494263:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758609.494279:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588655159168/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758609.494291:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758609.494295:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787758609.494298:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758609.494304:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11665 00000100:00080000:2.0:1787758609.494309:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758609.494316:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758609.494317:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758609/real 1787758609] req@ffff8c05b2e70e00 x1874588655158528/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787758609.494330:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758609.494353:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588655159296/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758609.494357:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588655158528/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758609.494379:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758609.494396:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758609.494399:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758609.494402:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758609.494404:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11665 00000100:00080000:0.0:1787758609.494413:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787758609.494414:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758609.494425:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758609.494426:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655158528:192.168.201.107@tcp:250:kworker.0 00000008:00100000:2.0:1787758609.494443:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758609.494456:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588655159424/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758609.494468:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655158656:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758609.494493:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758609.494504:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758609/real 1787758609] req@ffff8c05b2e72300 x1874588655158656/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758609.494512:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588655158656/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758609.494519:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758609.494522:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758609.494524:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655158656:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758609.494528:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655158784:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758609.494539:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758609.494546:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758609/real 1787758609] req@ffff8c05b2e72a00 x1874588655158784/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758609.494553:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588655158784/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758609.494559:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758609.494561:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758609.494562:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655158784:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758609.494567:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655158912:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758609.494580:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758609.494586:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758609/real 1787758609] req@ffff8c05b2e72680 x1874588655158912/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758609.494594:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588655158912/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758609.494600:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758609.494602:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758609.494603:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655158912:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758609.494607:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655159040:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758609.494619:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758609.494625:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758609/real 1787758609] req@ffff8c05b2e71180 x1874588655159040/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758609.494632:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588655159040/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758609.494638:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758609.494640:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758609.494641:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655159040:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758609.494645:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655159168:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758609.494656:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758609.494662:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758609/real 1787758609] req@ffff8c05b2e70000 x1874588655159168/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758609.494670:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588655159168/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758609.494675:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758609.494677:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758609.494678:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655159168:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758609.494682:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655159296:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758609.494691:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758609.494697:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758609/real 1787758609] req@ffff8c05b2e70a80 x1874588655159296/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758609.494704:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588655159296/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758609.494709:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758609.494711:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758609.494713:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655159296:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758609.494726:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655159424:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758609.494736:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758609.494742:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758609/real 1787758609] req@ffff8c05b2e73100 x1874588655159424/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758609.494749:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588655159424/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758609.494759:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758609.494762:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758609.494764:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655159424:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758609.621947:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758610.070206:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758610.645940:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758611.094093:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758611.669971:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758612.118337:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758612.693995:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758613.142039:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758613.717997:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758614.166150:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758614.614006:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758614.614016:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758614.614020:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758614.614033:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11671 00000100:00080000:3.0:1787758614.614045:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758614.614051:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758614.614096:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655159552/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758614.614125:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758614.614129:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758614.614131:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758614.614135:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11671 00000100:00080000:3.0:1787758614.614138:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758614.614141:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758614.614150:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758614.614161:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655159680/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758614.614169:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758614.614173:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758614.614174:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758614.614178:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11671 00000100:00080000:3.0:1787758614.614182:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787758614.614184:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655159552:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758614.614186:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758614.614195:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758614.614203:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655159808/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758614.614210:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758614.614213:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758614.614214:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758614.614217:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11671 00000100:00080000:3.0:1787758614.614229:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758614.614231:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758614.614235:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758614.614244:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655159936/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758614.614254:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758614.614257:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758614.614259:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758614.614261:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11671 00000100:00080000:3.0:1787758614.614264:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758614.614267:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758614.614272:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787758614.614281:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758614.614286:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655160064/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758614.614297:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758614.614301:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758614.614304:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758614.614309:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11671 00000100:00080000:3.0:1787758614.614314:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758614.614318:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758614.614324:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758614.614338:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655160192/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758614.614346:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787758614.614348:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758614/real 1787758614] req@ffff8c05a9f79500 x1874588655159552/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758614.614350:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758614.614353:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758614.614372:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11671 00000100:00080000:3.0:1787758614.614379:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758614.614384:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758614.614386:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655159552/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787758614.614392:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787758614.614400:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758614.614406:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758614.614408:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655160320/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758614.614413:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655159552:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758614.614424:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758614.614427:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758614.614428:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758614.614432:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11671 00000100:00080000:3.0:1787758614.614435:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758614.614438:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758614.614442:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758614.614443:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655159680:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758614.614451:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655160448/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787758614.614468:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758614.614490:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758614/real 1787758614] req@ffff8c05a9f7b800 x1874588655159680/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758614.614500:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655159680/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758614.614507:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758614.614509:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758614.614512:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655159680:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758614.614516:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655159808:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758614.614530:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758614.614537:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758614/real 1787758614] req@ffff8c05a9f78e00 x1874588655159808/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758614.614544:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655159808/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758614.614550:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758614.614552:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758614.614554:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655159808:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758614.614558:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655159936:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758614.614569:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758614.614574:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758614/real 1787758614] req@ffff8c05a9f7ad80 x1874588655159936/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758614.614582:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655159936/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758614.614588:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758614.614589:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758614.614591:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655159936:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758614.614595:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655160064:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758614.614606:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758614.614612:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758614/real 1787758614] req@ffff8c05a9f7b100 x1874588655160064/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758614.614619:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655160064/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758614.614624:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758614.614626:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758614.614628:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655160064:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758614.614631:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655160192:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758614.614650:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758614.614656:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758614/real 1787758614] req@ffff8c05a9f79f80 x1874588655160192/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758614.614663:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655160192/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758614.614668:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758614.614670:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758614.614672:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655160192:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758614.614675:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655160320:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758614.614685:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758614.614691:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758614/real 1787758614] req@ffff8c05a9f7bb80 x1874588655160320/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758614.614703:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655160320/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758614.614708:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758614.614710:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758614.614712:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655160320:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758614.614725:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655160448:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758614.614735:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758614.614740:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758614/real 1787758614] req@ffff8c05a9f79180 x1874588655160448/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758614.614747:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655160448/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758614.614752:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758614.614754:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758614.614756:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655160448:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758614.741976:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758615.190051:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758615.766002:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758616.214146:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758616.789960:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758617.238020:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758617.814022:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758618.262113:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758618.837961:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758619.286009:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758619.734041:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758619.734057:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758619.734062:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758619.734076:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11676 00000100:00080000:1.0:1787758619.734090:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758619.734099:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758619.734148:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655160576/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758619.734185:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758619.734189:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758619.734192:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758619.734196:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11676 00000100:00080000:1.0:1787758619.734201:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758619.734207:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758619.734219:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758619.734240:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588655160704/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758619.734254:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758619.734262:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787758619.734264:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655160576:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758619.734265:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758619.734282:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11676 00000100:00080000:1.0:1787758619.734287:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758619.734294:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758619.734300:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758619.734319:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655160832/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758619.734331:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758619.734336:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758619.734338:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758619.734344:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11676 00000800:00000100:0.0:1787758619.734350:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758619.734354:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758619.734363:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758619.734371:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758619.734387:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655160960/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758619.734398:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758619.734403:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758619.734406:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758619.734411:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11676 00000100:00080000:1.0:1787758619.734416:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787758619.734422:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758619/real 1787758619] req@ffff8c05b2d98380 x1874588655160576/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758619.734424:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758619.734439:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758619.734456:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655160576/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758619.734459:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655161088/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758619.734479:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758619.734493:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787758619.734495:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787758619.734501:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655160576:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758619.734505:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758619.734511:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758619.734517:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11676 00000100:00080000:1.0:1787758619.734522:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758619.734528:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758619.734532:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655160704:192.168.201.107@tcp:38:kworker.0 00000008:00100000:1.0:1787758619.734540:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758619.734563:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655161216/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787758619.734566:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758619.734584:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787758619.734592:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758619/real 1787758619] req@ffff8c05b2d9a300 x1874588655160704/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758619.734594:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758619.734599:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758619.734611:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11676 00000100:00080000:1.0:1787758619.734624:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787758619.734629:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588655160704/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758619.734635:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787758619.734644:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758619.734650:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000008:00100000:1.0:1787758619.734650:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758619.734657:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655160704:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758619.734667:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655160832:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758619.734679:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655161344/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787758619.734692:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758619.734697:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758619.734703:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787758619.734709:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758619/real 1787758619] req@ffff8c05b2d9bb80 x1874588655160832/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758619.734709:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758619.734721:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11676 00000100:00080000:1.0:1787758619.734733:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758619.734745:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758619.734753:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758619.734761:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655160832/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758619.734772:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655161472/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758619.734779:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758619.734808:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758619.734813:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655160832:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758619.734821:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655160960:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758619.734841:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758619.734854:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758619/real 1787758619] req@ffff8c05b2d9aa00 x1874588655160960/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758619.734867:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655160960/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758619.734906:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758619.734910:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758619.734912:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655160960:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758619.734917:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655161088:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758619.734935:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758619.734943:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758619/real 1787758619] req@ffff8c05b2d9b480 x1874588655161088/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758619.734951:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655161088/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758619.734957:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758619.734959:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758619.734961:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655161088:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758619.734975:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655161216:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758619.734987:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758619.734993:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758619/real 1787758619] req@ffff8c05b2d98700 x1874588655161216/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758619.735001:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655161216/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758619.735007:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758619.735010:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758619.735013:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655161216:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758619.735020:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655161344:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758619.735040:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758619.735052:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758619/real 1787758619] req@ffff8c05b2d99f80 x1874588655161344/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758619.735065:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655161344/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758619.735071:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758619.735073:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758619.735075:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655161344:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758619.735079:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655161472:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758619.735094:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758619.735101:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758619/real 1787758619] req@ffff8c05b2d98000 x1874588655161472/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758619.735108:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655161472/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758619.735117:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758619.735119:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758619.735121:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655161472:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758619.861999:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758620.310138:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758620.885960:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758621.334078:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758621.910010:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758622.358208:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758622.934038:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758623.382046:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758623.957971:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758624.406120:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758624.854022:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758624.854034:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758624.854039:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758624.854050:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11681 00000100:00080000:0.0:1787758624.854060:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758624.854067:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758624.854108:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655161600/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758624.854150:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758624.854155:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758624.854157:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758624.854161:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11681 00000100:00080000:0.0:1787758624.854164:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758624.854168:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758624.854182:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758624.854193:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655161600:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787758624.854197:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588655161728/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758624.854210:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758624.854213:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758624.854215:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758624.854218:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11681 00000100:00080000:0.0:1787758624.854221:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758624.854224:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758624.854227:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758624.854238:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588655161856/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758624.854247:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758624.854250:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758624.854251:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758624.854253:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11681 00000100:00080000:0.0:1787758624.854262:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758624.854265:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758624.854270:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787758624.854274:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758624.854287:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588655161984/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758624.854299:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758624.854303:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758624.854306:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758624.854310:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11681 00000100:00080000:0.0:1787758624.854315:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758624.854321:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758624.854325:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758624/real 1787758624] req@ffff8c05a9f75880 x1874588655161600/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787758624.854330:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758624.854346:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655161600/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758624.854352:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588655162112/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758624.854371:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758624.854372:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758624.854379:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758624.854382:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758624.854382:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787758624.854387:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11681 00000100:00080000:0.0:1787758624.854392:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758624.854392:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655161600:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758624.854399:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758624.854403:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758624.854415:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588655162240/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758624.854421:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655161728:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758624.854426:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758624.854430:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758624.854432:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758624.854436:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11681 00000100:00080000:0.0:1787758624.854443:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758624.854447:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:1.0:1787758624.854450:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:0.0:1787758624.854452:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758624.854463:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588655162368/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758624.854467:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758624/real 1787758624] req@ffff8c05a9f75500 x1874588655161728/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758624.854489:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758624.854494:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588655161728/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758624.854496:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758624.854502:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758624.854509:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758624.854511:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11681 00000100:00080000:0.0:1787758624.854518:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758624.854518:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787758624.854526:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758624.854528:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655161728:192.168.201.107@tcp:38:kworker.0 00000008:00100000:0.0:1787758624.854532:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758624.854540:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655161856:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758624.854547:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655162496/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787758624.854573:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758624.854586:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758624/real 1787758624] req@ffff8c05a9f74000 x1874588655161856/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758624.854595:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588655161856/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758624.854603:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758624.854606:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758624.854608:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655161856:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758624.854613:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655161984:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758624.854628:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758624.854635:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758624/real 1787758624] req@ffff8c05a9f75180 x1874588655161984/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758624.854645:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588655161984/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758624.854655:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758624.854659:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758624.854662:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655161984:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758624.854669:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655162112:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758624.854690:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758624.854716:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758624/real 1787758624] req@ffff8c05a9f76d80 x1874588655162112/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758624.854737:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588655162112/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758624.854746:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758624.854750:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758624.854753:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655162112:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758624.854772:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655162240:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758624.854795:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758624.854806:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758624/real 1787758624] req@ffff8c05a9f77100 x1874588655162240/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758624.854816:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588655162240/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758624.854830:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758624.854833:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758624.854836:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655162240:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758624.854842:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655162368:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758624.854861:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758624.854873:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758624/real 1787758624] req@ffff8c05a9f74e00 x1874588655162368/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758624.854904:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588655162368/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758624.854912:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758624.854916:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758624.854918:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655162368:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758624.854923:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655162496:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758624.854938:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758624.854946:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758624/real 1787758624] req@ffff8c05a9f74a80 x1874588655162496/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758624.854957:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655162496/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758624.854963:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758624.854965:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758624.854968:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655162496:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758624.981953:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758625.430065:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758626.005993:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758626.454168:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758627.029955:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758627.478042:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758628.054003:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758628.502128:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758629.077943:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758629.526199:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758629.974051:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758629.974066:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758629.974071:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758629.974085:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11686 00000100:00080000:1.0:1787758629.974098:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758629.974107:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758629.974157:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655162624/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758629.974189:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758629.974192:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758629.974195:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758629.974199:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11686 00000100:00080000:1.0:1787758629.974203:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758629.974208:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758629.974219:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758629.974235:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655162752/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758629.974241:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655162624:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758629.974249:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758629.974254:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758629.974256:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758629.974260:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11686 00000100:00080000:1.0:1787758629.974264:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758629.974269:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758629.974274:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758629.974289:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655162880/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758629.974300:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758629.974304:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758629.974306:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758629.974310:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11686 00000100:00080000:1.0:1787758629.974315:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758629.974320:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758629.974326:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787758629.974328:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758629.974342:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655163008/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758629.974353:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758629.974357:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758629.974360:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758629.974364:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11686 00000100:00080000:1.0:1787758629.974369:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758629.974374:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758629.974379:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758629.974393:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655163136/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758629.974399:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758629/real 1787758629] req@ffff8c05b2d99880 x1874588655162624/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758629.974407:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758629.974416:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758629.974421:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758629.974429:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11686 00000100:00080000:1.0:1787758629.974434:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787758629.974439:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655162624/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758629.974440:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758629.974449:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787758629.974463:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787758629.974468:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655163264/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758629.974472:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758629.974483:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655162624:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758629.974486:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758629.974493:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758629.974496:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758629.974502:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11686 00000100:00080000:1.0:1787758629.974506:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758629.974510:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758629.974516:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758629.974521:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655162752:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758629.974530:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655163392/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758629.974543:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758629.974547:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758629.974550:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787758629.974556:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758629.974557:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11686 00000100:00080000:1.0:1787758629.974566:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758629.974571:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758629.974578:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758629.974579:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758629/real 1787758629] req@ffff8c05b2d9b100 x1874588655162752/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758629.974595:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655163520/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758629.974627:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655162752/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758629.974640:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758629.974645:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758629.974650:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655162752:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758629.974660:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655162880:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758629.974684:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758629.974698:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758629/real 1787758629] req@ffff8c05b2d98000 x1874588655162880/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758629.974712:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655162880/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758629.974721:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758629.974725:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758629.974729:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655162880:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758629.974735:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655163008:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758629.974759:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758629.974773:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758629/real 1787758629] req@ffff8c05b2d99f80 x1874588655163008/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758629.974786:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655163008/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758629.974801:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758629.974804:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758629.974807:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655163008:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758629.974814:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655163136:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758629.974834:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758629.974846:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758629/real 1787758629] req@ffff8c05b2d98700 x1874588655163136/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758629.974858:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655163136/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758629.974866:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758629.974870:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758629.974874:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655163136:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758629.974904:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655163264:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758629.974927:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758629.974940:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758629/real 1787758629] req@ffff8c05b2d9b480 x1874588655163264/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758629.974955:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655163264/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758629.974964:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758629.974967:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758629.974972:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655163264:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758629.974993:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655163392:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758629.975015:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758629.975027:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758629/real 1787758629] req@ffff8c05b2d9aa00 x1874588655163392/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758629.975038:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655163392/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758629.975047:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758629.975050:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758629.975054:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655163392:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758629.975059:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655163520:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758629.975079:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758629.975091:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758629/real 1787758629] req@ffff8c05b2d9bb80 x1874588655163520/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758629.975104:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655163520/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758629.975113:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758629.975117:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758629.975120:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655163520:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758630.101971:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758630.550098:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758631.126009:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758631.574097:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758632.149981:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758632.598102:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758633.173995:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758633.622050:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758634.197977:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758634.646075:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758635.094000:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758635.094015:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758635.094020:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758635.094033:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11691 00000100:00080000:1.0:1787758635.094047:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758635.094056:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758635.094106:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588655163648/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758635.094141:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758635.094146:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758635.094148:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758635.094153:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11691 00000100:00080000:1.0:1787758635.094158:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758635.094162:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758635.094174:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758635.094194:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655163776/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758635.094204:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655163648:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758635.094211:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758635.094217:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758635.094220:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758635.094226:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11691 00000100:00080000:1.0:1787758635.094231:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758635.094236:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758635.094243:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758635.094259:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655163904/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758635.094272:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758635.094276:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758635.094278:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787758635.094283:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758635.094287:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11691 00000100:00080000:1.0:1787758635.094291:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758635.094297:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758635.094304:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758635.094320:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655164032/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758635.094330:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758635.094334:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758635.094336:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758635.094341:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11691 00000100:00100000:0.0:1787758635.094347:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758635/real 1787758635] req@ffff8c05b2d9a300 x1874588655163648/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758635.094347:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758635.094356:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758635.094367:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758635.094378:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588655163648/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758635.094386:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655164160/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758635.094394:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758635.094404:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787758635.094404:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787758635.094414:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655163648:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758635.094414:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758635.094420:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758635.094427:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11691 00000100:00080000:1.0:1787758635.094432:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758635.094437:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758635.094444:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758635.094447:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655163776:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758635.094462:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655164288/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758635.094475:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787758635.094477:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758635.094483:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758635.094486:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758635.094492:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11691 00000100:00100000:0.0:1787758635.094497:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758635/real 1787758635] req@ffff8c05b2d98380 x1874588655163776/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758635.094498:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758635.094507:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758635.094536:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655163776/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758635.094546:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758635.094549:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758635.094554:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655163776:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758635.094561:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655163904:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758635.094580:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758635.094592:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758635/real 1787758635] req@ffff8c05b2d9bb80 x1874588655163904/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758635.094610:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655163904/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758635.094619:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758635.094622:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758635.094626:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655163904:192.168.201.107@tcp:38:kworker.0 00000008:00100000:1.0:1787758635.094632:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758635.094645:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655164032:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758635.094653:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655164416/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758635.094665:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758635.094669:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1787758635.094675:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758635.094675:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758635.094684:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11691 00000100:00080000:1.0:1787758635.094690:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787758635.094692:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758635/real 1787758635] req@ffff8c05b2d9aa00 x1874588655164032/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758635.094698:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758635.094710:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758635.094716:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655164032/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758635.094726:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787758635.094728:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655164544/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758635.094733:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758635.094740:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655164032:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758635.094753:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655164160:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758635.094774:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758635.094785:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758635/real 1787758635] req@ffff8c05b2d9b480 x1874588655164160/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758635.094798:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655164160/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758635.094812:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758635.094816:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758635.094819:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655164160:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758635.094839:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655164288:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758635.094860:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758635.094921:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758635/real 1787758635] req@ffff8c05b2d98700 x1874588655164288/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758635.094936:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655164288/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758635.094945:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758635.094949:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758635.094953:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655164288:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758635.094960:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655164416:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758635.094982:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758635.094994:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758635/real 1787758635] req@ffff8c05b2d99f80 x1874588655164416/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758635.095008:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655164416/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758635.095016:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758635.095021:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758635.095026:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655164416:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758635.095034:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655164544:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758635.095056:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758635.095067:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758635/real 1787758635] req@ffff8c05b2d98000 x1874588655164544/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758635.095080:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655164544/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758635.095091:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758635.095095:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758635.095099:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655164544:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758635.221985:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758635.670034:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758636.245999:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758636.694104:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758637.269998:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758637.718107:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758638.293961:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758638.742066:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758639.317999:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758639.766045:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758640.213967:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758640.213978:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758640.213982:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758640.214002:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11696 00000100:00080000:1.0:1787758640.214012:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758640.214019:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758640.214069:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655164672/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758640.214101:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758640.214105:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758640.214107:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758640.214111:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11696 00000100:00080000:1.0:1787758640.214115:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758640.214119:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758640.214129:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758640.214133:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655164672:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758640.214146:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655164800/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758640.214156:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758640.214158:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758640.214160:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758640.214163:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11696 00000100:00080000:1.0:1787758640.214165:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758640.214168:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758640.214171:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758640.214180:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655164928/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758640.214188:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758640.214191:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758640.214192:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758640.214201:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11696 00000100:00080000:1.0:1787758640.214204:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758640.214207:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:0.0:1787758640.214211:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:1.0:1787758640.214213:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758640.214223:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655165056/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758640.214231:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758640.214234:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758640.214235:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758640.214239:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11696 00000100:00080000:1.0:1787758640.214244:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758640.214249:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758640.214255:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758640/real 1787758640] req@ffff8c05b2d9b100 x1874588655164672/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787758640.214255:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758640.214275:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655165184/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758640.214278:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655164672/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758640.214308:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758640.214309:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758640.214314:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758640.214317:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655164672:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758640.214320:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758640.214324:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758640.214329:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11696 00000100:00080000:1.0:1787758640.214332:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758640.214335:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758640.214339:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758640.214344:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655164800:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758640.214354:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655165312/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758640.214368:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758640.214373:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758640.214375:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787758640.214377:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758640.214382:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11696 00000100:00080000:1.0:1787758640.214388:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787758640.214393:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758640/real 1787758640] req@ffff8c05b2d99880 x1874588655164800/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758640.214396:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758640.214406:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758640.214410:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655164800/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758640.214418:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787758640.214420:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655165440/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758640.214437:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758640.214445:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655164800:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758640.214457:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655164928:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758640.214458:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758640.214465:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758640.214467:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758640.214471:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11696 00000100:00080000:1.0:1787758640.214475:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758640.214478:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758640.214483:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787758640.214484:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758640.214499:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655165568/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758640.214503:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758640/real 1787758640] req@ffff8c05b2d98000 x1874588655164928/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758640.214523:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655164928/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758640.214535:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758640.214540:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758640.214544:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655164928:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758640.214549:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655165056:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758640.214566:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758640.214574:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758640/real 1787758640] req@ffff8c05b2d99f80 x1874588655165056/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758640.214587:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655165056/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758640.214593:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758640.214595:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758640.214598:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655165056:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758640.214601:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655165184:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758640.214612:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758640.214618:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758640/real 1787758640] req@ffff8c05b2d98700 x1874588655165184/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758640.214626:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655165184/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758640.214631:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758640.214634:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758640.214636:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655165184:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758640.214648:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655165312:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758640.214659:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758640.214664:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758640/real 1787758640] req@ffff8c05b2d9b480 x1874588655165312/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758640.214672:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655165312/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758640.214681:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758640.214684:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758640.214686:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655165312:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758640.214689:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655165440:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758640.214699:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758640.214705:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758640/real 1787758640] req@ffff8c05b2d9aa00 x1874588655165440/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758640.214712:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655165440/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758640.214718:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758640.214720:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758640.214721:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655165440:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758640.214725:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655165568:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758640.214734:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758640.214740:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758640/real 1787758640] req@ffff8c05b2d9bb80 x1874588655165568/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758640.214747:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655165568/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758640.214753:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758640.214754:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758640.214757:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655165568:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758640.341942:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758640.790079:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758641.366018:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758641.814150:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758642.389951:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758642.838133:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758643.414039:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758643.862155:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758644.437984:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758644.886220:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758645.334038:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758645.334047:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758645.334051:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758645.334059:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11701 00000100:00080000:0.0:1787758645.334069:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758645.334076:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758645.334111:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655165696/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758645.334136:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758645.334139:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758645.334140:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758645.334143:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11701 00000100:00080000:0.0:1787758645.334153:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758645.334156:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758645.334164:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758645.334175:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655165824/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758645.334184:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655165696:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758645.334185:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758645.334193:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758645.334195:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758645.334200:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11701 00000100:00080000:0.0:1787758645.334202:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758645.334204:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758645.334208:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758645.334217:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655165952/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758645.334225:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758645.334227:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758645.334229:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758645.334231:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11701 00000100:00080000:0.0:1787758645.334246:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758645.334248:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758645.334252:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787758645.334259:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758645.334266:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588655166080/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758645.334278:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758645.334282:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758645.334284:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758645.334289:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11701 00000100:00080000:0.0:1787758645.334294:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758645.334299:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758645.334305:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758645.334312:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758645/real 1787758645] req@ffff8c05a9f76680 x1874588655165696/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758645.334319:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588655166208/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758645.334329:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758645.334346:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787758645.334347:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655165696/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758645.334348:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758645.334355:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11701 00000100:00080000:0.0:1787758645.334358:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758645.334365:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758645.334371:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787758645.334372:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758645.334378:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758645.334383:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655165696:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787758645.334385:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588655166336/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758645.334393:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758645.334396:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758645.334397:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758645.334400:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11701 00000100:00080000:0.0:1787758645.334402:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758645.334405:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758645.334409:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758645.334410:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655165824:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758645.334420:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655166464/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758645.334435:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787758645.334437:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758645.334439:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758645.334442:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758645.334447:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11701 00000100:00080000:0.0:1787758645.334451:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758645.334454:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758645/real 1787758645] req@ffff8c05a9f77800 x1874588655165824/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758645.334457:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758645.334464:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758645.334471:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655165824/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758645.334478:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655166592/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758645.334481:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758645.334489:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758645.334497:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655165824:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758645.334504:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655165952:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758645.334518:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758645.334526:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758645/real 1787758645] req@ffff8c05a9f74380 x1874588655165952/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758645.334533:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655165952/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758645.334538:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758645.334540:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758645.334542:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655165952:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758645.334545:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655166080:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758645.334556:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758645.334562:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758645/real 1787758645] req@ffff8c05a9f75c00 x1874588655166080/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758645.334569:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588655166080/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758645.334593:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758645.334596:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758645.334599:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655166080:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758645.334604:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655166208:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758645.334619:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758645.334628:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758645/real 1787758645] req@ffff8c05a9f76a00 x1874588655166208/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758645.334638:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588655166208/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758645.334646:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758645.334648:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758645.334651:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655166208:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758645.334656:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655166336:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758645.334670:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758645.334678:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758645/real 1787758645] req@ffff8c05a9f77b80 x1874588655166336/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758645.334688:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588655166336/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758645.334695:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758645.334698:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758645.334701:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655166336:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758645.334719:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655166464:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758645.334734:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758645.334743:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758645/real 1787758645] req@ffff8c05a9f76300 x1874588655166464/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758645.334754:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655166464/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758645.334761:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758645.334764:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758645.334767:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655166464:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758645.334772:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655166592:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758645.334786:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758645.334794:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758645/real 1787758645] req@ffff8c05a9f76680 x1874588655166592/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758645.334805:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655166592/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758645.334812:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758645.334815:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758645.334817:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655166592:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758645.461956:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758645.910122:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758646.485960:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758646.934068:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758647.509970:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758647.958034:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758648.533970:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758648.982163:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758649.557995:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758650.006081:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758650.453995:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758650.454005:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758650.454010:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758650.454027:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11706 00000100:00080000:1.0:1787758650.454039:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758650.454045:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758650.454096:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655166720/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758650.454130:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758650.454134:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758650.454136:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758650.454142:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11706 00000100:00080000:1.0:1787758650.454147:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758650.454152:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758650.454162:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758650.454166:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655166720:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758650.454177:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588655166848/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758650.454186:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758650.454189:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758650.454191:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758650.454194:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11706 00000100:00080000:1.0:1787758650.454207:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758650.454210:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758650.454213:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758650.454224:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655166976/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758650.454232:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758650.454235:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758650.454236:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758650.454238:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11706 00000100:00080000:1.0:1787758650.454258:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758650.454262:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:0.0:1787758650.454281:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:1.0:1787758650.454289:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758650.454301:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655167104/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758650.454309:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758650.454312:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758650.454313:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758650.454316:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11706 00000100:00080000:1.0:1787758650.454319:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758650.454322:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758650.454326:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758650.454341:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758650/real 1787758650] req@ffff8c05b2d98380 x1874588655166720/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758650.454341:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655167232/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758650.454371:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758650.454374:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758650.454375:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758650.454379:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11706 00000100:00100000:0.0:1787758650.454382:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655166720/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758650.454383:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758650.454390:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758650.454397:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787758650.454401:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758650.454407:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758650.454407:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655167360/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758650.454416:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655166720:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758650.454424:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758650.454427:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758650.454429:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758650.454432:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11706 00000100:00080000:1.0:1787758650.454435:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758650.454437:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758650.454440:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758650.454447:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655166848:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758650.454452:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655167488/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758650.454467:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758650.454470:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758650.454472:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787758650.454474:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758650.454478:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11706 00000100:00080000:1.0:1787758650.454482:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758650.454486:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758650.454490:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758650/real 1787758650] req@ffff8c05b2d9a300 x1874588655166848/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787758650.454490:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758650.454503:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655167616/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758650.454505:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588655166848/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758650.454530:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758650.454534:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758650.454537:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655166848:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758650.454541:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655166976:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758650.454556:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758650.454563:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758650/real 1787758650] req@ffff8c05b2d9bb80 x1874588655166976/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758650.454571:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655166976/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758650.454577:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758650.454579:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758650.454582:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655166976:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758650.454585:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655167104:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758650.454596:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758650.454602:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758650/real 1787758650] req@ffff8c05b2d9aa00 x1874588655167104/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758650.454610:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655167104/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758650.454615:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758650.454617:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758650.454618:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655167104:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758650.454622:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655167232:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758650.454633:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758650.454639:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758650/real 1787758650] req@ffff8c05b2d9b480 x1874588655167232/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758650.454647:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655167232/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758650.454653:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758650.454656:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758650.454658:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655167232:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758650.454664:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655167360:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758650.454683:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758650.454694:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758650/real 1787758650] req@ffff8c05b2d98700 x1874588655167360/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758650.454708:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655167360/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758650.454715:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758650.454718:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758650.454720:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655167360:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758650.454735:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655167488:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758650.454749:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758650.454756:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758650/real 1787758650] req@ffff8c05b2d99f80 x1874588655167488/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758650.454769:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655167488/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758650.454786:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758650.454788:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758650.454791:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655167488:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758650.454794:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655167616:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758650.454806:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758650.454813:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758650/real 1787758650] req@ffff8c05b2d98000 x1874588655167616/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758650.454820:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655167616/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758650.454826:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758650.454828:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758650.454830:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655167616:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758650.581977:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758651.030174:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758651.605973:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758652.054086:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758652.629968:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758653.078159:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758653.653960:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758654.102048:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758654.678006:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758655.126157:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758655.574048:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758655.574056:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787758655.574060:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758655.574069:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11711 00000100:00080000:2.0:1787758655.574080:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758655.574086:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758655.574121:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588655167744/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758655.574148:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758655.574150:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758655.574151:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758655.574166:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11711 00000100:00080000:2.0:1787758655.574169:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758655.574173:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758655.574181:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758655.574194:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588655167872/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758655.574202:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758655.574205:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758655.574206:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758655.574208:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11711 00000100:00080000:2.0:1787758655.574211:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758655.574213:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758655.574216:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758655.574223:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588655168000/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758655.574231:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758655.574235:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787758655.574238:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655167744:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787758655.574238:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758655.574243:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11711 00000100:00080000:2.0:1787758655.574247:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758655.574251:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758655.574254:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758655.574262:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588655168128/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758655.574269:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758655.574272:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758655.574273:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758655.574275:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11711 00000100:00080000:2.0:1787758655.574278:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758655.574281:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758655.574283:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758655.574291:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588655168256/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758655.574296:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758655.574298:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758655.574300:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758655.574303:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11711 00000100:00080000:2.0:1787758655.574305:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758655.574308:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:0.0:1787758655.574313:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:2.0:1787758655.574314:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758655.574324:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588655168384/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758655.574330:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758655.574333:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787758655.574334:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758655.574336:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11711 00000100:00080000:2.0:1787758655.574338:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758655.574341:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758655.574345:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758655.574353:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588655168512/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758655.574359:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758655.574361:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758655.574363:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787758655.574367:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758655/real 1787758655] req@ffff8c05b2e72d80 x1874588655167744/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758655.574368:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11711 00000100:00080000:2.0:1787758655.574373:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758655.574380:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758655.574386:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758655.574398:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588655168640/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758655.574402:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588655167744/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758655.574424:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758655.574427:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758655.574430:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655167744:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787758655.574436:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655167872:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758655.574453:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758655.574461:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758655/real 1787758655] req@ffff8c05b2e73100 x1874588655167872/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758655.574469:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588655167872/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758655.574475:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758655.574477:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758655.574479:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655167872:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758655.574498:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655168000:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758655.574508:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758655.574515:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758655/real 1787758655] req@ffff8c05b2e70a80 x1874588655168000/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758655.574522:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588655168000/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758655.574527:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758655.574529:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758655.574531:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655168000:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758655.574535:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655168128:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758655.574545:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758655.574551:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758655/real 1787758655] req@ffff8c05b2e70000 x1874588655168128/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758655.574558:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588655168128/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758655.574563:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758655.574565:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758655.574566:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655168128:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758655.574570:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655168256:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758655.574579:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758655.574585:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758655/real 1787758655] req@ffff8c05b2e71180 x1874588655168256/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758655.574592:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588655168256/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758655.574597:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758655.574599:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758655.574601:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655168256:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758655.574604:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655168384:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758655.574613:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758655.574619:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758655/real 1787758655] req@ffff8c05b2e72680 x1874588655168384/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758655.574627:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588655168384/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758655.574640:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758655.574642:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758655.574644:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655168384:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758655.574647:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655168512:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758655.574659:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758655.574665:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758655/real 1787758655] req@ffff8c05b2e72a00 x1874588655168512/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758655.574673:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588655168512/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758655.574678:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758655.574680:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758655.574682:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655168512:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758655.574685:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655168640:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758655.574694:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758655.574700:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758655/real 1787758655] req@ffff8c05b2e72300 x1874588655168640/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758655.574707:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588655168640/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758655.574716:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758655.574718:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758655.574720:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655168640:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758655.701964:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758656.150054:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758656.725967:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758657.174148:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758657.749949:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758658.198045:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758658.774012:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758659.222156:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758659.798005:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758660.246048:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758660.694010:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758660.694021:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758660.694025:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758660.694036:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11717 00000100:00080000:0.0:1787758660.694048:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758660.694055:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758660.694115:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655168768/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758660.694144:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758660.694147:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758660.694149:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758660.694152:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11717 00000100:00080000:0.0:1787758660.694155:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758660.694157:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758660.694166:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758660.694175:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588655168896/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758660.694184:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758660.694190:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787758660.694190:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655168768:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758660.694193:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758660.694201:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11717 00000100:00080000:0.0:1787758660.694204:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758660.694206:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758660.694210:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758660.694218:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588655169024/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758660.694225:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758660.694228:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758660.694230:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758660.694232:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11717 00000100:00080000:0.0:1787758660.694253:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758660.694256:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758660.694259:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787758660.694266:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758660.694269:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588655169152/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758660.694277:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758660.694280:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758660.694282:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758660.694284:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11717 00000100:00080000:0.0:1787758660.694288:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758660.694291:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758660.694294:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758660.694302:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588655169280/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758660.694308:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758660.694310:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787758660.694313:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758660/real 1787758660] req@ffff8c05a9f74a80 x1874588655168768/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758660.694314:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758660.694321:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11717 00000100:00080000:0.0:1787758660.694326:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758660.694333:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758660.694337:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758660.694342:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655168768/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758660.694347:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588655169408/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758660.694368:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758660.694368:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758660.694376:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758660.694378:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787758660.694379:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758660.694386:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11717 00000100:00100000:1.0:1787758660.694386:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655168768:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758660.694392:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758660.694396:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758660.694399:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758660.694405:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588655169536/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758660.694414:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758660.694415:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655168896:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758660.694421:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758660.694424:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758660.694428:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11717 00000100:00080000:0.0:1787758660.694432:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758660.694435:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758660.694439:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787758660.694440:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758660.694451:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655169664/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758660.694456:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758660/real 1787758660] req@ffff8c05a9f74e00 x1874588655168896/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758660.694473:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588655168896/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758660.694480:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758660.694483:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758660.694485:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655168896:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758660.694491:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655169024:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758660.694503:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758660.694510:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758660/real 1787758660] req@ffff8c05a9f77100 x1874588655169024/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758660.694518:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588655169024/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758660.694523:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758660.694524:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758660.694526:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655169024:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758660.694530:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655169152:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758660.694542:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758660.694548:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758660/real 1787758660] req@ffff8c05a9f76d80 x1874588655169152/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758660.694555:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588655169152/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758660.694560:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758660.694562:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758660.694564:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655169152:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758660.694568:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655169280:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758660.694579:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758660.694586:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758660/real 1787758660] req@ffff8c05a9f75180 x1874588655169280/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758660.694593:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588655169280/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758660.694599:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758660.694600:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758660.694602:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655169280:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758660.694605:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655169408:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758660.694615:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758660.694621:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758660/real 1787758660] req@ffff8c05a9f74000 x1874588655169408/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758660.694628:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588655169408/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758660.694633:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758660.694635:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758660.694637:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655169408:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758660.694650:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655169536:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758660.694660:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758660.694666:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758660/real 1787758660] req@ffff8c05a9f75500 x1874588655169536/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758660.694672:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588655169536/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758660.694682:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758660.694684:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758660.694686:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655169536:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758660.694690:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655169664:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758660.694699:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758660.694704:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758660/real 1787758660] req@ffff8c05a9f75880 x1874588655169664/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758660.694711:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655169664/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758660.694716:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758660.694718:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758660.694720:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655169664:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758660.821980:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758661.270216:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758661.845970:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758662.294098:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758662.870003:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758663.318182:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758663.894072:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758664.342123:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758664.918012:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758665.366164:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758665.814241:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758665.814252:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758665.814256:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758665.814267:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11722 00000100:00080000:1.0:1787758665.814278:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758665.814285:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758665.814334:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655169792/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758665.814363:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758665.814366:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758665.814368:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758665.814371:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11722 00000100:00080000:1.0:1787758665.814373:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758665.814375:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758665.814385:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758665.814398:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655169920/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758665.814408:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758665.814412:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758665.814415:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758665.814419:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11722 00000100:00080000:1.0:1787758665.814424:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758665.814428:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758665.814434:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758665.814446:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655170048/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758665.814455:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787758665.814458:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655169792:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758665.814458:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758665.814461:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758665.814467:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11722 00000100:00080000:1.0:1787758665.814471:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758665.814473:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758665.814477:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758665.814483:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655170176/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758665.814489:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758665.814492:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758665.814493:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758665.814495:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11722 00000100:00080000:1.0:1787758665.814497:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758665.814500:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758665.814502:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758665.814509:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655170304/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758665.814514:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758665.814517:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758665.814518:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758665.814520:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11722 00000100:00080000:1.0:1787758665.814523:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758665.814525:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758665.814529:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758665.814540:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655170432/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758665.814554:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758665.814558:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758665.814560:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758665.814564:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11722 00000100:00080000:1.0:1787758665.814578:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758665.814582:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758665.814587:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758665.814598:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655170560/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758665.814604:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758665.814606:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758665.814608:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758665.814610:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11722 00000100:00080000:1.0:1787758665.814613:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758665.814616:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758665.814618:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758665.814624:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655170688/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758665.814667:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655169920:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758665.814748:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655170048:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758665.814996:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655170176:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758665.815026:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655170304:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758665.815048:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655170432:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758665.815066:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655170560:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758665.815084:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655170688:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758665.942019:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758666.390136:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758666.966024:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758667.414138:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758667.989973:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758668.438133:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787758668.886203:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.107:988 00000400:00000100:2.0:1787758668.886250:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.107@tcp at host 192.168.201.107 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787758668.886270:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787758668.886320:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787758668.886331:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787758668.886339:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787758668.886350:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787758668.886359:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787758668.886376:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:0.0:1787758668.886385:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758665/real 1787758668] req@ffff8c05b2d98000 x1874588655170048/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787758668.886387:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:0.0:1787758668.886425:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655170048/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758668.886437:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758668.886448:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758668.886453:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655170048:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758668.886462:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758665/real 1787758668] req@ffff8c05b2d99f80 x1874588655170176/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758668.886475:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655170176/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758668.886483:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758668.886488:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758668.886491:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655170176:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758668.886499:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758665/real 1787758668] req@ffff8c05b2d98700 x1874588655170304/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758668.886511:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655170304/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758668.886521:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758668.886525:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758668.886528:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655170304:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758668.886535:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758665/real 1787758668] req@ffff8c05b2d9b480 x1874588655170432/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758668.886548:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655170432/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758668.886556:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758668.886559:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758668.886563:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655170432:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758668.886569:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758665/real 1787758668] req@ffff8c05b2d9aa00 x1874588655170560/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758668.886583:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655170560/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758668.886591:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758668.886595:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758668.886598:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655170560:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758668.886605:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758665/real 1787758668] req@ffff8c05b2d9bb80 x1874588655170688/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758668.886617:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655170688/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758668.886626:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758668.886629:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758668.886633:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655170688:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758668.886639:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758665/real 1787758668] req@ffff8c05b2d99880 x1874588655169792/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758668.886654:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655169792/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758668.886663:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758668.886666:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758668.886669:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655169792:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787758668.886677:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758665/real 1787758668] req@ffff8c05b2d9b100 x1874588655169920/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758668.886690:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655169920/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758668.886698:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758668.886701:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758668.886704:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655169920:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758669.014043:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758669.462181:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758670.037967:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758670.486109:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758670.934032:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758670.934047:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758670.934052:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758670.934066:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11727 00000100:00080000:1.0:1787758670.934079:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758670.934087:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758670.934131:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588655170816/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758670.934165:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758670.934170:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758670.934171:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758670.934176:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11727 00000100:00080000:1.0:1787758670.934179:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758670.934183:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758670.934193:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758670.934210:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655170944/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758670.934222:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758670.934226:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758670.934228:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758670.934233:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11727 00000100:00100000:0.0:1787758670.934238:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655170816:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758670.934238:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758670.934244:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758670.934252:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758670.934268:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655171072/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758670.934280:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758670.934284:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758670.934287:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758670.934291:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11727 00000100:00080000:1.0:1787758670.934294:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758670.934299:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758670.934304:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758670.934317:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655171200/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758670.934328:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787758670.934332:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758670.934334:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758670.934338:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758670.934343:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11727 00000100:00080000:1.0:1787758670.934348:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758670.934353:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758670.934359:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758670.934372:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655171328/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758670.934382:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758670.934386:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758670.934391:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787758670.934392:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758670/real 1787758670] req@ffff8c05b2d9a300 x1874588655170816/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758670.934397:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11727 00000100:00080000:1.0:1787758670.934407:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758670.934414:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758670.934420:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758670.934423:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588655170816/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758670.934436:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655171456/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758670.934439:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758670.934451:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787758670.934456:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787758670.934459:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655170816:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758670.934462:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758670.934467:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787758670.934473:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655170944:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758670.934477:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11727 00000100:00080000:1.0:1787758670.934484:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758670.934489:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758670.934494:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787758670.934506:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758670.934511:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655171584/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758670.934523:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787758670.934525:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758670/real 1787758670] req@ffff8c05b2d98380 x1874588655170944/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758670.934528:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758670.934533:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758670.934544:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11727 00000100:00080000:1.0:1787758670.934550:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787758670.934552:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655170944/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758670.934558:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758670.934569:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787758670.934570:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758670.934578:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758670.934583:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655170944:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758670.934588:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655171712/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758670.934618:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655171072:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758670.934644:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758670.934656:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758670/real 1787758670] req@ffff8c05b2d9b100 x1874588655171072/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758670.934669:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655171072/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758670.934679:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758670.934683:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758670.934687:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655171072:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758670.934694:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655171200:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758670.934714:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758670.934726:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758670/real 1787758670] req@ffff8c05b2d99880 x1874588655171200/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758670.934738:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655171200/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758670.934749:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758670.934753:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758670.934763:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655171200:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758670.934770:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655171328:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758670.934789:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758670.934801:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758670/real 1787758670] req@ffff8c05b2d9bb80 x1874588655171328/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758670.934815:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655171328/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758670.934824:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758670.934828:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758670.934831:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655171328:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758670.934837:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655171456:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758670.934858:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758670.934871:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758670/real 1787758670] req@ffff8c05b2d9aa00 x1874588655171456/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758670.934945:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655171456/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758670.934955:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758670.934960:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758670.934964:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655171456:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758670.934971:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655171584:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758670.934994:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758670.935006:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758670/real 1787758670] req@ffff8c05b2d9b480 x1874588655171584/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758670.935018:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655171584/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758670.935028:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758670.935032:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758670.935037:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655171584:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758670.935044:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655171712:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758670.935065:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758670.935076:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758670/real 1787758670] req@ffff8c05b2d98700 x1874588655171712/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758670.935091:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655171712/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758670.935101:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758670.935104:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758670.935108:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655171712:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758671.061989:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758671.510126:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758672.085961:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758672.534044:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758673.109970:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758673.558116:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758674.133998:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758674.582069:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758675.157990:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758675.606106:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758676.054000:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758676.054011:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758676.054016:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758676.054027:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11732 00000100:00080000:1.0:1787758676.054038:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758676.054045:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758676.054085:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655171840/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758676.054109:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758676.054112:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758676.054113:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758676.054117:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11732 00000100:00080000:1.0:1787758676.054119:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758676.054122:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758676.054131:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758676.054144:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655171968/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758676.054152:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758676.054155:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758676.054156:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758676.054159:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11732 00000100:00080000:1.0:1787758676.054161:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758676.054163:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758676.054166:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758676.054170:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655171840:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758676.054178:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655172096/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758676.054187:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758676.054189:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758676.054190:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758676.054192:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11732 00000100:00080000:1.0:1787758676.054195:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758676.054198:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758676.054201:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758676.054209:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655172224/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758676.054219:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758676.054222:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758676.054224:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758676.054229:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11732 00000100:00080000:1.0:1787758676.054244:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758676.054249:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758676.054255:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787758676.054258:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758676.054281:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655172352/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758676.054292:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758676.054294:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758676.054296:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758676.054300:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11732 00000100:00080000:1.0:1787758676.054304:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758676.054309:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758676.054310:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758676/real 1787758676] req@ffff8c05b2d99f80 x1874588655171840/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787758676.054313:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758676.054327:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655172480/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758676.054334:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758676.054337:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758676.054339:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787758676.054342:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655171840/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758676.054345:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11732 00000100:00080000:1.0:1787758676.054349:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758676.054355:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758676.054362:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787758676.054367:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787758676.054372:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655172608/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758676.054377:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787758676.054386:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787758676.054387:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655171840:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758676.054392:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758676.054396:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758676.054400:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11732 00000100:00100000:0.0:1787758676.054404:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655171968:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758676.054405:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758676.054412:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758676.054416:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758676.054425:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655172736/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787758676.054440:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758676.054460:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758676/real 1787758676] req@ffff8c05b2d98000 x1874588655171968/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758676.054476:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655171968/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758676.054487:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758676.054491:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758676.054494:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655171968:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758676.054522:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655172096:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758676.054543:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758676.054554:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758676/real 1787758676] req@ffff8c05b2d98700 x1874588655172096/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758676.054567:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655172096/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758676.054576:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758676.054580:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758676.054583:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655172096:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758676.054590:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655172224:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758676.054619:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758676.054629:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758676/real 1787758676] req@ffff8c05b2d9b480 x1874588655172224/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758676.054642:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655172224/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758676.054650:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758676.054654:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758676.054658:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655172224:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758676.054664:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655172352:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758676.054682:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758676.054691:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758676/real 1787758676] req@ffff8c05b2d9aa00 x1874588655172352/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758676.054711:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655172352/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758676.054719:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758676.054723:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758676.054726:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655172352:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758676.054733:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655172480:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758676.054748:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758676.054758:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758676/real 1787758676] req@ffff8c05b2d9bb80 x1874588655172480/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758676.054771:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655172480/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758676.054779:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758676.054783:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758676.054786:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655172480:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758676.054809:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655172608:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758676.054829:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758676.054840:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758676/real 1787758676] req@ffff8c05b2d99880 x1874588655172608/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758676.054852:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655172608/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758676.054868:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758676.054871:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758676.054898:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655172608:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758676.054905:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655172736:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758676.054925:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758676.054936:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758676/real 1787758676] req@ffff8c05b2d9b100 x1874588655172736/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758676.054948:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655172736/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758676.054957:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758676.054961:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758676.054965:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655172736:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758676.182029:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758676.630002:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758677.205954:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758677.654129:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758678.230018:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758678.678088:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758679.253982:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758679.702139:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758680.277985:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758680.726043:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758681.174029:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758681.174042:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787758681.174047:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758681.174060:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11737 00000100:00080000:2.0:1787758681.174072:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758681.174081:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758681.174129:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588655172864/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758681.174170:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758681.174174:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758681.174176:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758681.174180:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11737 00000100:00080000:2.0:1787758681.174183:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758681.174188:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758681.174199:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758681.174213:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588655172992/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758681.174226:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787758681.174232:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655172864:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787758681.174232:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758681.174236:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758681.174246:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11737 00000100:00080000:2.0:1787758681.174250:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758681.174254:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758681.174260:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758681.174273:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588655173120/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758681.174285:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758681.174289:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758681.174291:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758681.174295:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11737 00000100:00080000:2.0:1787758681.174300:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758681.174304:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758681.174310:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787758681.174320:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758681.174324:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588655173248/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758681.174335:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758681.174340:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758681.174343:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758681.174347:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11737 00000100:00080000:2.0:1787758681.174352:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758681.174356:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758681.174362:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758681.174377:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588655173376/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758681.174383:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758681/real 1787758681] req@ffff8c05b2e70e00 x1874588655172864/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758681.174395:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758681.174400:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758681.174403:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787758681.174410:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588655172864/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758681.174410:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11737 00000100:00080000:2.0:1787758681.174418:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758681.174427:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787758681.174428:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787758681.174433:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758681.174447:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588655173504/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758681.174450:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758681.174462:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655172864:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787758681.174478:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758681.174482:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787758681.174483:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758681.174489:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11737 00000100:00080000:2.0:1787758681.174493:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758681.174497:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758681.174502:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655172992:192.168.201.107@tcp:38:kworker.0 00000008:00100000:2.0:1787758681.174504:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758681.174521:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588655173632/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758681.174532:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758681.174537:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758681.174539:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758681.174543:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11737 00000100:00080000:2.0:1787758681.174548:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787758681.174549:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787758681.174557:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758681.174564:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758681.174571:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758681/real 1787758681] req@ffff8c05b2e70700 x1874588655172992/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758681.174579:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588655173760/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758681.174591:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588655172992/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758681.174605:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758681.174609:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758681.174612:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655172992:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758681.174620:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655173120:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758681.174643:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758681.174653:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758681/real 1787758681] req@ffff8c05b2e73480 x1874588655173120/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758681.174661:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588655173120/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758681.174667:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758681.174670:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758681.174672:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655173120:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758681.174676:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655173248:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758681.174688:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758681.174695:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758681/real 1787758681] req@ffff8c05b2e71500 x1874588655173248/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758681.174716:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588655173248/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758681.174723:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758681.174726:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758681.174728:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655173248:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758681.174732:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655173376:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758681.174748:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758681.174756:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758681/real 1787758681] req@ffff8c05b2e73b80 x1874588655173376/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758681.174763:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588655173376/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758681.174769:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758681.174771:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758681.174773:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655173376:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758681.174776:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655173504:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758681.174786:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758681.174792:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758681/real 1787758681] req@ffff8c05b2e71c00 x1874588655173504/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758681.174799:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588655173504/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758681.174805:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758681.174807:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758681.174810:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655173504:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758681.174823:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655173632:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758681.174833:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758681.174839:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758681/real 1787758681] req@ffff8c05b2e71880 x1874588655173632/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758681.174846:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588655173632/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758681.174851:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758681.174853:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758681.174855:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655173632:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758681.174858:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655173760:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758681.174868:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758681.174903:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758681/real 1787758681] req@ffff8c05b2e70380 x1874588655173760/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758681.174912:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588655173760/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758681.174917:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758681.174919:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758681.174922:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655173760:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758681.301973:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758681.750154:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758682.325989:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758682.774115:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758683.349955:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758683.798067:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758684.373985:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758684.822063:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758685.397953:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758685.846076:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758686.294051:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758686.294062:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758686.294067:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758686.294078:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11742 00000100:00080000:0.0:1787758686.294089:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758686.294097:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758686.294141:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655173888/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758686.294172:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758686.294175:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758686.294177:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758686.294180:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11742 00000100:00080000:0.0:1787758686.294183:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758686.294186:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758686.294194:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758686.294204:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588655174016/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758686.294212:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758686.294215:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758686.294216:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758686.294220:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11742 00000100:00080000:0.0:1787758686.294223:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758686.294226:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758686.294232:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655173888:192.168.201.107@tcp:250:kworker.0 00000008:00100000:0.0:1787758686.294234:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758686.294252:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588655174144/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758686.294266:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758686.294270:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758686.294272:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758686.294277:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11742 00000100:00080000:0.0:1787758686.294282:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758686.294286:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758686.294292:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758686.294301:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588655174272/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787758686.294329:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758686.294334:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758686.294338:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758686.294339:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758686.294368:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11742 00000100:00080000:0.0:1787758686.294379:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758686.294398:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758686.294403:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758686/real 1787758686] req@ffff8c05a9f75880 x1874588655173888/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787758686.294413:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758686.294434:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655173888/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758686.294448:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758686.294453:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758686.294458:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655173888:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758686.294468:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655174016:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758686.294499:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758686.294513:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588655174400/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758686.294519:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758686/real 1787758686] req@ffff8c05a9f75500 x1874588655174016/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758686.294525:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758686.294533:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758686.294536:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758686.294545:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11742 00000100:00080000:0.0:1787758686.294552:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758686.294558:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758686.294562:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588655174016/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787758686.294563:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787758686.294576:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787758686.294579:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588655174528/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758686.294582:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758686.294588:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655174016:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758686.294606:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758686.294610:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758686.294624:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758686.294630:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11742 00000100:00100000:1.0:1787758686.294631:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655174144:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758686.294639:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758686.294646:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758686.294653:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787758686.294661:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758686.294667:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588655174656/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758686.294678:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758686.294679:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758686/real 1787758686] req@ffff8c05a9f74000 x1874588655174144/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758686.294683:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758686.294687:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758686.294695:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11742 00000100:00080000:0.0:1787758686.294702:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758686.294705:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758686.294709:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758686.294709:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588655174144/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758686.294721:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787758686.294722:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655174784/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758686.294728:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758686.294735:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655174144:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758686.294745:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655174272:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758686.294771:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758686.294784:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758686/real 1787758686] req@ffff8c05a9f75180 x1874588655174272/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758686.294798:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588655174272/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758686.294807:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758686.294811:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758686.294814:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655174272:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758686.294822:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655174400:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758686.294855:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758686.294869:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758686/real 1787758686] req@ffff8c05a9f76d80 x1874588655174400/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758686.294932:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588655174400/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758686.294942:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758686.294945:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758686.294949:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655174400:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758686.294971:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655174528:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758686.294989:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758686.294998:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758686/real 1787758686] req@ffff8c05a9f77100 x1874588655174528/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758686.295006:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588655174528/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758686.295012:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758686.295014:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758686.295016:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655174528:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758686.295032:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655174656:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758686.295062:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758686.295075:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758686/real 1787758686] req@ffff8c05a9f74e00 x1874588655174656/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758686.295087:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588655174656/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758686.295098:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758686.295100:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758686.295102:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655174656:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758686.295106:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655174784:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758686.295121:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758686.295128:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758686/real 1787758686] req@ffff8c05a9f74a80 x1874588655174784/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758686.295135:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655174784/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758686.295140:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758686.295142:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758686.295144:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655174784:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758686.421998:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758686.870065:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758687.446015:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758687.894096:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758688.469985:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758688.918019:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758689.493986:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758689.942133:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758690.517976:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758690.966023:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758691.413953:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758691.413964:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758691.413969:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758691.413982:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11747 00000100:00080000:0.0:1787758691.413996:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758691.414005:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758691.414050:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655174912/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758691.414077:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758691.414080:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758691.414082:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758691.414085:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11747 00000100:00080000:0.0:1787758691.414088:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758691.414091:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758691.414099:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758691.414108:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655175040/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758691.414115:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758691.414119:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758691.414120:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758691.414123:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11747 00000100:00080000:0.0:1787758691.414125:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758691.414127:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758691.414131:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758691.414138:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588655175168/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758691.414148:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758691.414152:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758691.414154:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758691.414159:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11747 00000100:00080000:0.0:1787758691.414165:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758691.414165:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655174912:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758691.414175:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758691.414182:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758691.414196:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588655175296/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758691.414205:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758691.414208:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758691.414209:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758691.414212:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11747 00000100:00080000:0.0:1787758691.414229:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758691.414232:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758691.414236:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787758691.414240:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758691.414247:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588655175424/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758691.414255:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758691.414258:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758691.414259:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758691.414263:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11747 00000100:00080000:0.0:1787758691.414265:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758691.414267:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758691.414270:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758691.414278:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655175552/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758691.414284:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758691.414287:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758691.414288:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758691.414291:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758691/real 1787758691] req@ffff8c05a9f76680 x1874588655174912/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758691.414294:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11747 00000100:00080000:0.0:1787758691.414299:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758691.414306:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758691.414311:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758691.414318:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655174912/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758691.414325:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655175680/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758691.414330:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758691.414339:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787758691.414341:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758691.414345:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655174912:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758691.414348:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758691.414352:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758691.414353:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655175040:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758691.414364:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11747 00000100:00080000:0.0:1787758691.414369:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758691.414373:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758691.414378:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787758691.414378:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758691.414391:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758691/real 1787758691] req@ffff8c05a9f76300 x1874588655175040/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758691.414396:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655175808/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758691.414410:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655175040/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758691.414420:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758691.414423:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758691.414426:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655175040:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758691.414430:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655175168:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758691.414443:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758691.414449:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758691/real 1787758691] req@ffff8c05a9f77b80 x1874588655175168/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758691.414457:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588655175168/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758691.414462:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758691.414464:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758691.414466:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655175168:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758691.414489:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655175296:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758691.414501:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758691.414507:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758691/real 1787758691] req@ffff8c05a9f76a00 x1874588655175296/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758691.414514:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588655175296/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758691.414519:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758691.414521:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758691.414539:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655175296:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758691.414544:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655175424:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758691.414557:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758691.414564:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758691/real 1787758691] req@ffff8c05a9f75c00 x1874588655175424/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758691.414571:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588655175424/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758691.414576:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758691.414579:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758691.414580:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655175424:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758691.414584:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655175552:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758691.414593:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758691.414599:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758691/real 1787758691] req@ffff8c05a9f74380 x1874588655175552/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758691.414611:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655175552/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758691.414616:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758691.414617:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758691.414619:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655175552:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758691.414623:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655175680:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758691.414633:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758691.414639:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758691/real 1787758691] req@ffff8c05a9f77800 x1874588655175680/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758691.414646:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655175680/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758691.414651:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758691.414653:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758691.414654:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655175680:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758691.414658:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655175808:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758691.414667:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758691.414673:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758691/real 1787758691] req@ffff8c05a9f74a80 x1874588655175808/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758691.414680:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655175808/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758691.414689:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758691.414691:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758691.414701:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655175808:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758691.541962:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758691.990149:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758692.565957:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758693.014050:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758693.589984:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758694.038160:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758694.613945:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758695.061994:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758695.637954:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758696.086054:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758696.533970:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758696.533982:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758696.533988:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758696.534002:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11752 00000100:00080000:0.0:1787758696.534015:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758696.534023:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758696.534066:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588655175936/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758696.534096:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758696.534100:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758696.534102:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758696.534107:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11752 00000100:00080000:0.0:1787758696.534112:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758696.534117:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758696.534128:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758696.534137:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655175936:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787758696.534146:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588655176064/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758696.534159:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758696.534163:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758696.534165:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758696.534169:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11752 00000100:00080000:0.0:1787758696.534173:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758696.534178:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758696.534183:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758696.534196:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588655176192/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758696.534208:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758696.534214:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1787758696.534215:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758696.534219:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758696.534225:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11752 00000100:00080000:0.0:1787758696.534230:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758696.534235:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758696.534241:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758696.534253:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588655176320/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758696.534265:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758696.534269:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758696/real 1787758696] req@ffff8c05a9f74e00 x1874588655175936/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758696.534270:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758696.534274:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758696.534283:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11752 00000100:00080000:0.0:1787758696.534291:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758696.534296:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758696.534301:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758696.534304:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588655175936/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758696.534315:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588655176448/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758696.534320:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758696.534330:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787758696.534331:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758696.534337:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787758696.534339:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655175936:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758696.534340:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758696.534350:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11752 00000100:00080000:0.0:1787758696.534355:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758696.534359:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758696.534363:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758696.534375:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655176064:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758696.534376:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588655176576/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758696.534394:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758696.534398:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758696.534400:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758696.534404:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11752 00000100:00080000:0.0:1787758696.534408:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787758696.534414:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758696.534419:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758696.534425:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758696.534450:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758696/real 1787758696] req@ffff8c05a9f77100 x1874588655176064/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758696.534455:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655176704/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758696.534470:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758696.534478:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758696.534483:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758696.534488:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11752 00000100:00100000:1.0:1787758696.534493:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588655176064/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758696.534494:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758696.534503:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758696.534512:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787758696.534513:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758696.534521:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758696.534526:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588655176832/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758696.534529:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655176064:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758696.534544:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655176192:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758696.534571:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758696.534584:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758696/real 1787758696] req@ffff8c05a9f76d80 x1874588655176192/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758696.534597:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588655176192/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758696.534606:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758696.534608:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758696.534610:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655176192:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758696.534615:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655176320:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758696.534632:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758696.534640:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758696/real 1787758696] req@ffff8c05a9f75180 x1874588655176320/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758696.534647:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588655176320/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758696.534652:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758696.534654:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758696.534656:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655176320:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758696.534660:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655176448:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758696.534671:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758696.534677:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758696/real 1787758696] req@ffff8c05a9f74000 x1874588655176448/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758696.534684:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588655176448/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758696.534689:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758696.534690:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758696.534692:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655176448:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758696.534704:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655176576:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758696.534715:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758696.534721:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758696/real 1787758696] req@ffff8c05a9f75500 x1874588655176576/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758696.534727:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588655176576/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758696.534733:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758696.534734:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758696.534736:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655176576:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758696.534739:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655176704:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758696.534748:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758696.534754:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758696/real 1787758696] req@ffff8c05a9f75880 x1874588655176704/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758696.534766:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655176704/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758696.534771:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758696.534772:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758696.534774:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655176704:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758696.534777:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655176832:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758696.534786:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758696.534791:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758696/real 1787758696] req@ffff8c05a9f74e00 x1874588655176832/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758696.534798:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588655176832/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758696.534803:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758696.534805:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758696.534807:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655176832:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758696.661940:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758697.110046:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758697.685997:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758698.134127:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758698.709997:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758699.158103:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758699.734023:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758700.182139:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758700.758001:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758701.206073:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758701.653988:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758701.654000:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758701.654005:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758701.654015:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11758 00000100:00080000:1.0:1787758701.654026:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758701.654032:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758701.654092:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655176960/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758701.654116:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758701.654119:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758701.654120:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758701.654123:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11758 00000100:00080000:1.0:1787758701.654126:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758701.654132:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758701.654140:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758701.654151:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588655177088/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758701.654159:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758701.654163:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787758701.654165:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655176960:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758701.654165:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758701.654172:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11758 00000100:00080000:1.0:1787758701.654176:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758701.654178:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758701.654181:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758701.654193:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655177216/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758701.654200:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758701.654203:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758701.654205:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758701.654208:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11758 00000100:00080000:1.0:1787758701.654229:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758701.654232:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758701.654235:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787758701.654244:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758701.654246:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655177344/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758701.654265:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758701.654267:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758701.654269:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758701.654271:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11758 00000100:00080000:1.0:1787758701.654274:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758701.654276:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758701.654280:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758701.654287:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655177472/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758701.654293:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758701.654295:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758701.654296:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758701.654299:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11758 00000100:00080000:1.0:1787758701.654301:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758701.654304:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758701.654308:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758701.654318:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655177600/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758701.654324:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758701/real 1787758701] req@ffff8c05b2d98380 x1874588655176960/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758701.654324:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758701.654333:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758701.654336:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758701.654354:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11758 00000100:00080000:1.0:1787758701.654360:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758701.654366:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758701.654371:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758701.654373:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655176960/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758701.654384:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787758701.654388:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655177728/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758701.654391:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758701.654398:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655176960:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758701.654414:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758701.654418:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758701.654419:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758701.654423:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11758 00000100:00080000:1.0:1787758701.654427:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787758701.654429:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655177088:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758701.654434:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758701.654440:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758701.654452:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655177856/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787758701.654460:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758701.654475:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758701/real 1787758701] req@ffff8c05b2d9a300 x1874588655177088/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758701.654485:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588655177088/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758701.654492:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758701.654494:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758701.654496:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655177088:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758701.654501:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655177216:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758701.654513:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758701.654520:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758701/real 1787758701] req@ffff8c05b2d9b100 x1874588655177216/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758701.654528:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655177216/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758701.654533:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758701.654534:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758701.654536:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655177216:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758701.654540:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655177344:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758701.654551:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758701.654556:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758701/real 1787758701] req@ffff8c05b2d99880 x1874588655177344/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758701.654568:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655177344/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758701.654576:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758701.654580:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758701.654583:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655177344:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758701.654590:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655177472:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758701.654611:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758701.654633:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758701/real 1787758701] req@ffff8c05b2d9bb80 x1874588655177472/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758701.654641:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655177472/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758701.654647:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758701.654649:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758701.654652:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655177472:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758701.654656:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655177600:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758701.654669:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758701.654675:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758701/real 1787758701] req@ffff8c05b2d9aa00 x1874588655177600/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758701.654682:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655177600/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758701.654687:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758701.654689:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758701.654692:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655177600:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758701.654696:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655177728:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758701.654706:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758701.654712:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758701/real 1787758701] req@ffff8c05b2d9b480 x1874588655177728/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758701.654719:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655177728/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758701.654724:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758701.654726:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758701.654728:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655177728:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758701.654742:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655177856:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758701.654753:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758701.654759:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758701/real 1787758701] req@ffff8c05b2d98700 x1874588655177856/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758701.654767:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655177856/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758701.654772:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758701.654773:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758701.654775:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655177856:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758701.782000:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758702.230189:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758702.805982:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758703.254184:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758703.829979:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758704.278099:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758704.853954:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758705.302108:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758705.877975:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758706.326123:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758706.774008:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758706.774020:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787758706.774026:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758706.774039:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11763 00000100:00080000:2.0:1787758706.774053:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758706.774063:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758706.774108:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73800 x1874588655177984/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758706.774137:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758706.774143:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758706.774144:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758706.774148:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11763 00000100:00080000:2.0:1787758706.774151:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758706.774156:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758706.774165:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758706.774177:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588655178112/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758706.774203:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758706.774207:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787758706.774209:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655177984:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787758706.774209:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758706.774218:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11763 00000100:00080000:2.0:1787758706.774222:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758706.774226:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758706.774230:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758706.774243:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588655178240/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758706.774278:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758706.774282:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758706.774284:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758706.774288:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11763 00000100:00080000:2.0:1787758706.774292:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758706.774296:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758706.774300:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758706.774310:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588655178368/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758706.774318:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787758706.774324:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787758706.774325:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758706.774328:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758706.774333:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11763 00000100:00080000:2.0:1787758706.774336:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758706.774339:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758706.774342:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758706.774351:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588655178496/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758706.774358:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758706.774361:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758706.774362:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758706.774365:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11763 00000100:00080000:2.0:1787758706.774367:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758706.774370:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758706.774374:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758706.774375:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758706/real 1787758706] req@ffff8c05b2e73800 x1874588655177984/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758706.774385:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588655178624/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758706.774397:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758706.774400:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787758706.774403:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758706.774405:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73800 x1874588655177984/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758706.774409:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11763 00000100:00080000:2.0:1787758706.774414:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758706.774420:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758706.774426:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787758706.774431:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787758706.774435:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588655178752/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758706.774442:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787758706.774446:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758706.774449:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787758706.774453:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655177984:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787758706.774453:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758706.774458:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11763 00000100:00080000:2.0:1787758706.774462:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758706.774466:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758706.774469:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758706.774476:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588655178880/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758706.774492:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655178112:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758706.774523:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758706.774540:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758706/real 1787758706] req@ffff8c05b2e70380 x1874588655178112/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758706.774556:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588655178112/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758706.774567:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758706.774571:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758706.774575:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655178112:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758706.774582:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655178240:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758706.774605:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758706.774619:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758706/real 1787758706] req@ffff8c05b2e71880 x1874588655178240/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758706.774634:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588655178240/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758706.774649:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758706.774653:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758706.774657:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655178240:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758706.774663:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655178368:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758706.774687:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758706.774701:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758706/real 1787758706] req@ffff8c05b2e71c00 x1874588655178368/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758706.774713:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588655178368/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758706.774722:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758706.774726:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758706.774730:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655178368:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758706.774736:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655178496:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758706.774758:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758706.774771:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758706/real 1787758706] req@ffff8c05b2e73b80 x1874588655178496/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758706.774800:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588655178496/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758706.774812:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758706.774825:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758706.774832:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655178496:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758706.774841:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655178624:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758706.774870:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758706.774925:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758706/real 1787758706] req@ffff8c05b2e71500 x1874588655178624/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758706.774939:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588655178624/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758706.774949:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758706.774953:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758706.774968:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655178624:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758706.774979:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655178752:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758706.775023:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758706.775038:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758706/real 1787758706] req@ffff8c05b2e73480 x1874588655178752/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758706.775070:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588655178752/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758706.775080:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758706.775084:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758706.775088:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655178752:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758706.775094:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655178880:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758706.775117:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758706.775130:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758706/real 1787758706] req@ffff8c05b2e70700 x1874588655178880/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758706.775145:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588655178880/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758706.775154:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758706.775157:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758706.775177:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655178880:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758706.902012:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758707.350095:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758707.925981:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758708.374118:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758708.949971:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758709.398168:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758709.973953:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758710.422217:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758710.998020:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758711.446059:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758711.894032:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758711.894045:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758711.894050:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758711.894063:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11768 00000100:00080000:1.0:1787758711.894074:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758711.894082:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758711.894128:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655179008/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758711.894161:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758711.894181:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758711.894184:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758711.894188:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11768 00000100:00080000:1.0:1787758711.894192:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758711.894197:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758711.894206:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655179008:192.168.201.107@tcp:250:kworker.0 00000008:00100000:1.0:1787758711.894207:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758711.894228:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655179136/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758711.894238:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758711.894242:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758711.894243:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758711.894247:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11768 00000100:00080000:1.0:1787758711.894265:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758711.894269:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758711.894274:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787758711.894275:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758711.894290:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655179264/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758711.894302:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758711.894305:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758711.894306:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758711.894310:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11768 00000100:00080000:1.0:1787758711.894314:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758711.894318:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758711.894322:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758711/real 1787758711] req@ffff8c05b2d98000 x1874588655179008/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787758711.894326:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758711.894343:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655179392/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758711.894346:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655179008/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758711.894372:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758711.894373:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758711.894380:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787758711.894381:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1787758711.894385:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655179008:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758711.894385:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758711.894395:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11768 00000100:00080000:1.0:1787758711.894399:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758711.894404:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758711.894409:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758711.894413:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655179136:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758711.894421:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655179520/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787758711.894434:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758711.894434:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758711.894440:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758711.894442:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787758711.894448:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758711/real 1787758711] req@ffff8c05b2d99f80 x1874588655179136/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758711.894449:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11768 00000100:00080000:1.0:1787758711.894455:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758711.894463:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758711.894466:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655179136/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787758711.894470:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787758711.894476:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758711.894480:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758711.894482:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655179136:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758711.894488:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655179648/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758711.894490:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655179264:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758711.894505:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787758711.894510:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758711.894513:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758711.894517:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787758711.894521:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758711/real 1787758711] req@ffff8c05b2d98700 x1874588655179264/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758711.894525:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11768 00000100:00080000:1.0:1787758711.894532:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787758711.894538:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655179264/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758711.894540:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758711.894549:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787758711.894551:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758711.894557:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758711.894560:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655179264:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758711.894569:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655179392:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758711.894570:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655179776/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758711.894582:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787758711.894585:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758711.894589:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758711.894592:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787758711.894596:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758711/real 1787758711] req@ffff8c05b2d9b480 x1874588655179392/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758711.894599:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11768 00000100:00080000:1.0:1787758711.894606:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787758711.894614:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655179392/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758711.894615:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758711.894624:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787758711.894625:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758711.894631:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758711.894633:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655179392:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758711.894641:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655179904/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758711.894648:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655179520:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758711.894667:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758711.894675:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758711/real 1787758711] req@ffff8c05b2d9aa00 x1874588655179520/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758711.894684:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655179520/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758711.894690:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758711.894692:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758711.894694:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655179520:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758711.894697:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655179648:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758711.894707:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758711.894713:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758711/real 1787758711] req@ffff8c05b2d9bb80 x1874588655179648/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758711.894720:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655179648/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758711.894725:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758711.894727:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758711.894728:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655179648:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758711.894731:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655179776:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758711.894740:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758711.894746:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758711/real 1787758711] req@ffff8c05b2d99880 x1874588655179776/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758711.894753:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655179776/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758711.894761:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758711.894764:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758711.894768:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655179776:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758711.894791:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655179904:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758711.894813:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758711.894825:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758711/real 1787758711] req@ffff8c05b2d9b100 x1874588655179904/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758711.894838:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655179904/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758711.894847:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758711.894850:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758711.894854:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655179904:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758712.021953:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758712.470301:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758713.045972:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758713.494046:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758714.069962:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758714.518097:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758715.094005:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758715.542117:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758716.117950:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758716.566144:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758717.013976:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758717.013992:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758717.013998:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758717.014014:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11773 00000100:00080000:1.0:1787758717.014026:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758717.014035:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758717.014091:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588655180032/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758717.014130:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758717.014133:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758717.014134:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758717.014138:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11773 00000100:00080000:1.0:1787758717.014141:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758717.014145:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758717.014154:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758717.014178:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655180032:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758717.014179:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655180160/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758717.014193:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758717.014196:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758717.014197:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758717.014199:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11773 00000100:00080000:1.0:1787758717.014201:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758717.014204:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758717.014207:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758717.014217:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655180288/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758717.014224:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758717.014227:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758717.014229:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758717.014232:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11773 00000100:00080000:1.0:1787758717.014245:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758717.014249:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758717.014255:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787758717.014256:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758717.014274:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655180416/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758717.014293:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758717.014297:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758717.014299:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787758717.014302:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758717/real 1787758717] req@ffff8c05b2d9a300 x1874588655180032/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758717.014305:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11773 00000100:00080000:1.0:1787758717.014311:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758717.014318:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758717.014334:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758717.014338:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588655180032/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758717.014349:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787758717.014350:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655180544/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758717.014356:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758717.014364:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655180032:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758717.014373:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758717.014378:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758717.014381:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758717.014386:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11773 00000100:00100000:0.0:1787758717.014390:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655180160:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758717.014392:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758717.014400:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758717.014405:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787758717.014414:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758717.014421:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655180672/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758717.014428:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758717/real 1787758717] req@ffff8c05b2d98380 x1874588655180160/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758717.014434:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758717.014441:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758717.014444:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758717.014449:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11773 00000100:00080000:1.0:1787758717.014453:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787758717.014457:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655180160/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758717.014470:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758717.014477:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787758717.014479:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758717.014484:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758717.014488:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655180800/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758717.014489:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655180160:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758717.014502:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655180288:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758717.014505:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758717.014508:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758717.014510:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758717.014513:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11773 00000100:00080000:1.0:1787758717.014520:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787758717.014523:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758717.014525:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758717.014531:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758717.014536:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758717/real 1787758717] req@ffff8c05b2d9b100 x1874588655180288/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758717.014542:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655180928/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758717.014554:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655180288/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758717.014566:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758717.014569:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758717.014572:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655180288:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758717.014578:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655180416:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758717.014591:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758717.014597:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758717/real 1787758717] req@ffff8c05b2d99880 x1874588655180416/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758717.014610:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655180416/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758717.014616:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758717.014618:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758717.014620:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655180416:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758717.014623:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655180544:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758717.014633:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758717.014639:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758717/real 1787758717] req@ffff8c05b2d9bb80 x1874588655180544/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758717.014647:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655180544/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758717.014652:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758717.014654:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758717.014655:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655180544:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758717.014666:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655180672:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758717.014677:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758717.014683:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758717/real 1787758717] req@ffff8c05b2d9aa00 x1874588655180672/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758717.014690:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655180672/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758717.014707:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758717.014709:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758717.014711:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655180672:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758717.014715:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655180800:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758717.014726:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758717.014733:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758717/real 1787758717] req@ffff8c05b2d9b480 x1874588655180800/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758717.014740:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655180800/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758717.014746:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758717.014747:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758717.014749:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655180800:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758717.014753:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655180928:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758717.014763:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758717.014769:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758717/real 1787758717] req@ffff8c05b2d98700 x1874588655180928/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758717.014776:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655180928/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758717.014782:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758717.014783:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758717.014786:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655180928:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758717.142032:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758717.590127:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758718.166020:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758718.614187:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758719.190018:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758719.638120:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758720.214006:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758720.662158:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758721.238017:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758721.686132:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758722.134050:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758722.134062:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758722.134066:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758722.134078:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11778 00000100:00080000:0.0:1787758722.134090:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758722.134097:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758722.134143:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655181056/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758722.134174:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758722.134177:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758722.134178:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758722.134182:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11778 00000100:00080000:0.0:1787758722.134186:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758722.134189:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758722.134199:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758722.134211:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655181184/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758722.134219:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758722.134226:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758722.134228:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758722.134228:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655181056:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758722.134237:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11778 00000100:00080000:0.0:1787758722.134245:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758722.134250:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758722.134254:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758722.134265:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655181312/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758722.134272:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758722.134275:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758722.134277:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758722.134281:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11778 00000100:00080000:0.0:1787758722.134303:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758722.134311:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:1.0:1787758722.134313:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:0.0:1787758722.134326:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758722.134344:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588655181440/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758722.134352:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758722.134355:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758722.134357:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758722.134361:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11778 00000100:00080000:0.0:1787758722.134364:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758722.134371:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758722/real 1787758722] req@ffff8c05a9f74a80 x1874588655181056/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758722.134372:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758722.134383:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758722.134400:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588655181568/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758722.134409:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655181056/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758722.134415:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758722.134430:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758722.134432:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758722.134433:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758722.134441:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11778 00000100:00080000:1.0:1787758722.134446:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787758722.134450:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758722.134459:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655181056:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758722.134462:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758722.134467:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758722.134481:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588655181696/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758722.134491:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758722.134494:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758722.134496:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758722.134501:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11778 00000100:00080000:0.0:1787758722.134505:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758722.134506:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655181184:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758722.134513:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758722.134517:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758722.134530:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655181824/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758722.134541:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787758722.134542:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758722.134551:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758722.134553:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758722.134559:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11778 00000100:00080000:0.0:1787758722.134565:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758722.134566:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758722/real 1787758722] req@ffff8c05a9f77800 x1874588655181184/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758722.134572:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758722.134584:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758722.134592:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655181184/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758722.134600:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655181952/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758722.134607:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758722.134626:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758722.134634:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655181184:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758722.134643:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655181312:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758722.134664:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758722.134673:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758722/real 1787758722] req@ffff8c05a9f74380 x1874588655181312/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758722.134681:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655181312/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758722.134686:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758722.134688:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758722.134690:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655181312:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758722.134695:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655181440:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758722.134708:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758722.134714:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758722/real 1787758722] req@ffff8c05a9f75c00 x1874588655181440/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758722.134723:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588655181440/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758722.134728:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758722.134730:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758722.134732:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655181440:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758722.134736:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655181568:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758722.134747:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758722.134754:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758722/real 1787758722] req@ffff8c05a9f76a00 x1874588655181568/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758722.134761:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588655181568/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758722.134766:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758722.134768:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758722.134770:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655181568:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758722.134774:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655181696:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758722.134797:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758722.134805:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758722/real 1787758722] req@ffff8c05a9f77b80 x1874588655181696/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758722.134818:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588655181696/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758722.134824:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758722.134826:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758722.134828:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655181696:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758722.134842:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655181824:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758722.134853:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758722.134860:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758722/real 1787758722] req@ffff8c05a9f76300 x1874588655181824/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758722.134868:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655181824/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758722.134894:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758722.134896:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758722.134898:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655181824:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758722.134902:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655181952:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758722.134915:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758722.134923:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758722/real 1787758722] req@ffff8c05a9f76680 x1874588655181952/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758722.134931:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655181952/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758722.134941:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758722.134943:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758722.134946:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655181952:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758722.261962:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758722.710181:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758723.285987:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758723.734055:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758724.310023:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758724.758217:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758725.334027:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758725.782076:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758726.358052:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758726.806172:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758727.254015:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758727.254026:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758727.254030:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758727.254042:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11783 00000100:00080000:0.0:1787758727.254053:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758727.254061:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758727.254125:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655182080/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758727.254151:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758727.254154:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758727.254156:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758727.254159:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11783 00000100:00080000:0.0:1787758727.254163:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758727.254166:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758727.254174:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758727.254185:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655182208/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758727.254193:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758727.254197:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758727.254198:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758727.254201:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11783 00000100:00080000:0.0:1787758727.254204:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758727.254206:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758727.254212:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655182080:192.168.201.107@tcp:250:kworker.0 00000008:00100000:0.0:1787758727.254213:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758727.254223:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588655182336/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758727.254230:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758727.254233:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758727.254234:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758727.254238:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11783 00000100:00080000:0.0:1787758727.254240:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758727.254243:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758727.254246:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758727.254254:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588655182464/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758727.254261:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758727.254264:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758727.254265:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758727.254267:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11783 00000100:00080000:0.0:1787758727.254273:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758727.254275:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758727.254279:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787758727.254286:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758727.254288:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588655182592/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758727.254296:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758727.254298:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758727.254300:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758727.254302:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11783 00000100:00080000:0.0:1787758727.254311:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758727.254314:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758727.254317:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758727.254324:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655182720/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758727.254330:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758727.254334:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758727.254336:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758727.254341:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11783 00000100:00080000:0.0:1787758727.254346:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758727.254346:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758727/real 1787758727] req@ffff8c05a9f76680 x1874588655182080/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758727.254355:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758727.254364:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758727.254368:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655182080/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758727.254380:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655182848/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758727.254380:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758727.254388:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758727.254395:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655182080:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758727.254397:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758727.254403:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758727.254405:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758727.254409:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11783 00000100:00080000:0.0:1787758727.254413:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758727.254415:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758727.254419:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758727.254424:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655182208:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758727.254430:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655182976/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787758727.254449:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758727.254461:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758727/real 1787758727] req@ffff8c05a9f76300 x1874588655182208/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758727.254470:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655182208/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758727.254477:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758727.254480:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758727.254483:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655182208:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758727.254488:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655182336:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758727.254503:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758727.254516:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758727/real 1787758727] req@ffff8c05a9f77b80 x1874588655182336/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758727.254531:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588655182336/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758727.254540:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758727.254544:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758727.254547:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655182336:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758727.254554:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655182464:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758727.254574:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758727.254583:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758727/real 1787758727] req@ffff8c05a9f76a00 x1874588655182464/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758727.254591:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588655182464/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758727.254597:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758727.254599:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758727.254601:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655182464:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758727.254605:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655182592:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758727.254616:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758727.254622:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758727/real 1787758727] req@ffff8c05a9f75c00 x1874588655182592/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758727.254630:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588655182592/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758727.254636:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758727.254637:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758727.254639:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655182592:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758727.254642:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655182720:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758727.254653:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758727.254658:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758727/real 1787758727] req@ffff8c05a9f74380 x1874588655182720/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758727.254665:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655182720/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758727.254671:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758727.254672:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758727.254674:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655182720:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758727.254677:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655182848:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758727.254687:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758727.254692:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758727/real 1787758727] req@ffff8c05a9f77800 x1874588655182848/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758727.254704:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655182848/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758727.254711:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758727.254714:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758727.254717:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655182848:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758727.254738:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655182976:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758727.254759:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758727.254770:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758727/real 1787758727] req@ffff8c05a9f74a80 x1874588655182976/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758727.254779:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655182976/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758727.254784:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758727.254786:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758727.254788:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655182976:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758727.382017:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758727.830064:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758728.406055:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758728.854166:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758729.430021:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758729.878050:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758730.453987:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758730.902131:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758731.477947:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758731.926032:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758732.373981:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758732.373991:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787758732.373996:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758732.374008:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11788 00000100:00080000:2.0:1787758732.374018:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758732.374025:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758732.374066:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588655183104/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758732.374096:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758732.374100:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758732.374108:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758732.374113:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11788 00000100:00080000:2.0:1787758732.374118:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758732.374124:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758732.374136:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758732.374151:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588655183232/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758732.374156:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655183104:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787758732.374166:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758732.374170:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758732.374171:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758732.374175:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11788 00000100:00080000:2.0:1787758732.374178:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758732.374181:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758732.374185:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758732.374194:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588655183360/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758732.374202:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758732.374205:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758732.374207:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758732.374209:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11788 00000100:00080000:2.0:1787758732.374211:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758732.374214:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758732.374217:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758732.374224:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588655183488/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758732.374230:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758732.374233:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758732.374235:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758732.374238:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11788 00000100:00080000:2.0:1787758732.374241:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758732.374245:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758732.374251:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758732.374265:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588655183616/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758732.374277:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758732.374281:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758732.374284:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758732.374289:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11788 00000100:00080000:2.0:1787758732.374295:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758732.374298:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655183232:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787758732.374303:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758732.374317:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758732.374334:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588655183744/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758732.374339:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655183360:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787758732.374349:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758732.374352:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787758732.374354:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758732.374358:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11788 00000100:00080000:2.0:1787758732.374361:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758732.374363:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758732.374369:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758732.374375:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655183488:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758732.374382:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588655183872/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758732.374390:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758732.374393:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758732.374395:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758732.374398:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11788 00000100:00080000:2.0:1787758732.374401:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758732.374403:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655183616:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787758732.374407:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758732.374413:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758732.374424:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588655184000/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758732.374432:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655183744:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758732.374465:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655183872:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758732.374522:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655184000:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758732.502013:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758732.950128:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758733.525970:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758733.974059:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758734.549966:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758734.998131:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787758735.446289:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.107:988 00000400:00000100:1.0:1787758735.446354:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.107@tcp at host 192.168.201.107 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787758735.446376:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787758735.446436:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787758735.446450:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787758735.446461:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787758735.446470:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787758735.446482:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787758735.446498:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787758735.446508:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:2.0:1787758735.446647:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758732/real 1787758735] req@ffff8c05b2e73100 x1874588655184000/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758735.446678:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588655184000/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758735.446692:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758735.446698:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758735.446703:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655184000:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758735.446712:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758732/real 1787758735] req@ffff8c05b2e70000 x1874588655183744/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758735.446726:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588655183744/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758735.446735:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758735.446739:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758735.446742:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655183744:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758735.446771:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758732/real 1787758735] req@ffff8c05b2e70a80 x1874588655183872/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758735.446785:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588655183872/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758735.446794:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758735.446798:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758735.446802:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655183872:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758735.446810:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758732/real 1787758735] req@ffff8c05b2e72300 x1874588655183232/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758735.446822:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588655183232/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758735.446830:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758735.446834:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758735.446838:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655183232:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758735.446857:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758732/real 1787758735] req@ffff8c05b2e72a00 x1874588655183360/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758735.446872:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588655183360/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758735.446912:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758735.446917:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758735.446921:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655183360:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758735.446928:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758732/real 1787758735] req@ffff8c05b2e72680 x1874588655183488/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758735.446941:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588655183488/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758735.446950:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758735.446954:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758735.446957:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655183488:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758735.446965:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758732/real 1787758735] req@ffff8c05b2e71180 x1874588655183616/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758735.446976:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588655183616/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758735.446984:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758735.446987:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758735.446990:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655183616:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758735.447002:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758732/real 1787758735] req@ffff8c05b2e70e00 x1874588655183104/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758735.447017:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588655183104/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758735.447027:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758735.447031:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758735.447035:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655183104:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787758735.573974:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758736.022050:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758736.598003:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758737.046138:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758737.493980:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758737.493991:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787758737.493995:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758737.494007:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11793 00000100:00080000:2.0:1787758737.494016:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758737.494023:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758737.494062:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588655184128/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758737.494113:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758737.494118:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758737.494120:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758737.494125:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11793 00000100:00080000:2.0:1787758737.494129:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758737.494133:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758737.494144:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758737.494163:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588655184256/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758737.494176:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758737.494180:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758737.494182:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758737.494188:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11793 00000100:00080000:2.0:1787758737.494192:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758737.494201:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655184128:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787758737.494203:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758737.494210:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758737.494231:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588655184384/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758737.494244:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758737.494248:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758737.494252:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758737.494257:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11793 00000100:00080000:2.0:1787758737.494264:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758737.494269:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758737.494276:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758737.494292:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588655184512/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787758737.494299:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787758737.494315:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758737.494320:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758737.494322:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758737.494329:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11793 00000100:00080000:2.0:1787758737.494335:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758737.494341:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758737.494348:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758737.494362:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758737/real 1787758737] req@ffff8c05b2e70e00 x1874588655184128/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758737.494367:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588655184640/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758737.494404:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758737.494411:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787758737.494413:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588655184128/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758737.494416:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758737.494432:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11793 00000100:00080000:2.0:1787758737.494445:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758737.494456:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758737.494469:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787758737.494475:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758737.494484:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758737.494490:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588655184768/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758737.494497:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655184128:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787758737.494523:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758737.494524:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655184256:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787758737.494538:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787758737.494543:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758737.494552:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11793 00000100:00080000:2.0:1787758737.494558:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758737.494564:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:1.0:1787758737.494571:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:2.0:1787758737.494572:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758737.494596:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588655184896/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758737.494598:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758737/real 1787758737] req@ffff8c05b2e71180 x1874588655184256/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758737.494638:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588655184256/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758737.494651:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758737.494660:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758737.494661:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758737.494664:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758737.494679:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787758737.494679:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11793 00000100:00080000:2.0:1787758737.494693:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758737.494697:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655184256:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787758737.494705:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758737.494731:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758737.494747:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588655185024/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758737.494757:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655184384:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758737.494801:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758737.494823:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758737/real 1787758737] req@ffff8c05b2e72680 x1874588655184384/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758737.494835:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588655184384/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758737.494843:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758737.494847:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758737.494850:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655184384:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758737.494858:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655184512:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758737.494934:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758737.494950:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758737/real 1787758737] req@ffff8c05b2e72a00 x1874588655184512/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758737.494965:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588655184512/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758737.494975:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758737.494981:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758737.494985:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655184512:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758737.494994:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655184640:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758737.495021:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758737.495037:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758737/real 1787758737] req@ffff8c05b2e72300 x1874588655184640/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758737.495051:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588655184640/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758737.495060:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758737.495063:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758737.495067:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655184640:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758737.495073:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655184768:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758737.495094:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758737.495106:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758737/real 1787758737] req@ffff8c05b2e70a80 x1874588655184768/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758737.495118:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588655184768/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758737.495126:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758737.495129:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758737.495132:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655184768:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758737.495138:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655184896:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758737.495159:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758737.495171:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758737/real 1787758737] req@ffff8c05b2e70000 x1874588655184896/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758737.495184:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588655184896/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758737.495193:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758737.495197:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758737.495200:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655184896:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758737.495222:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655185024:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758737.495244:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758737.495256:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758737/real 1787758737] req@ffff8c05b2e73100 x1874588655185024/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758737.495269:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588655185024/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758737.495286:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758737.495289:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758737.495293:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655185024:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758737.621999:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758738.070019:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758738.646037:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758739.094106:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758739.670012:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758740.118049:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758740.694042:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758741.142129:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758741.717978:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758742.166140:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758742.614010:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758742.614021:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787758742.614027:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758742.614039:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11799 00000100:00080000:2.0:1787758742.614051:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758742.614058:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758742.614101:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588655185152/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758742.614134:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758742.614138:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758742.614140:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758742.614143:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11799 00000100:00080000:2.0:1787758742.614146:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758742.614149:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758742.614157:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758742.614174:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588655185280/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758742.614183:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758742.614186:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758742.614189:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758742.614196:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11799 00000100:00100000:0.0:1787758742.614198:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655185152:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787758742.614199:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758742.614209:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758742.614216:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758742.614231:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588655185408/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758742.614240:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758742.614243:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758742.614245:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758742.614250:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11799 00000100:00080000:2.0:1787758742.614254:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758742.614258:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758742.614263:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758742.614274:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588655185536/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758742.614283:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758742.614289:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758742.614293:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787758742.614294:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787758742.614311:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11799 00000100:00080000:2.0:1787758742.614316:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758742.614320:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758742.614326:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758742.614336:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588655185664/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758742.614346:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758742.614351:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758742.614353:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758742.614361:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11799 00000100:00080000:2.0:1787758742.614367:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787758742.614368:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758742/real 1787758742] req@ffff8c05b2e72d80 x1874588655185152/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758742.614381:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758742.614396:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758742.614409:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588655185152/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758742.614410:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588655185792/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758742.614480:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758742.614482:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758742.614484:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787758742.614491:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758742.614499:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787758742.614502:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11799 00000100:00100000:0.0:1787758742.614510:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655185152:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787758742.614519:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758742.614527:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758742.614535:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758742.614548:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655185280:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758742.614555:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588655185920/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758742.614572:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787758742.614581:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787758742.614582:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758742.614589:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758742.614598:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11799 00000100:00100000:0.0:1787758742.614601:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758742/real 1787758742] req@ffff8c05b2e73100 x1874588655185280/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758742.614609:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787758742.614618:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588655185280/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758742.614620:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758742.614644:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787758742.614657:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758742.614663:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758742.614668:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655185280:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758742.614672:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588655186048/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758742.614685:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655185408:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758742.614716:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758742.614736:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758742/real 1787758742] req@ffff8c05b2e70000 x1874588655185408/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758742.614749:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588655185408/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758742.614755:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758742.614758:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758742.614760:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655185408:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758742.614764:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655185536:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758742.614776:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758742.614783:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758742/real 1787758742] req@ffff8c05b2e70a80 x1874588655185536/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758742.614790:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588655185536/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758742.614800:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758742.614802:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758742.614804:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655185536:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758742.614807:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655185664:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758742.614817:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758742.614823:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758742/real 1787758742] req@ffff8c05b2e72300 x1874588655185664/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758742.614830:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588655185664/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758742.614835:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758742.614837:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758742.614839:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655185664:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758742.614842:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655185792:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758742.614853:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758742.614859:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758742/real 1787758742] req@ffff8c05b2e72a00 x1874588655185792/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758742.614866:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588655185792/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758742.614871:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758742.614873:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758742.614902:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655185792:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758742.614926:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655185920:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758742.614945:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758742.614953:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758742/real 1787758742] req@ffff8c05b2e72680 x1874588655185920/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758742.614961:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588655185920/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758742.614966:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758742.614968:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758742.614970:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655185920:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758742.614974:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655186048:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758742.614984:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758742.614990:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758742/real 1787758742] req@ffff8c05b2e71180 x1874588655186048/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758742.614997:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588655186048/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758742.615002:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758742.615004:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758742.615006:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655186048:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758742.742001:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758743.190155:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758743.765998:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758744.214107:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758744.790011:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758745.238125:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758745.814010:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758746.262078:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758746.837956:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758747.286109:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758747.733976:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758747.733986:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787758747.733991:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758747.734002:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11804 00000100:00080000:2.0:1787758747.734013:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758747.734020:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758747.734070:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588655186176/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758747.734095:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758747.734098:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758747.734100:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758747.734103:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11804 00000100:00080000:2.0:1787758747.734106:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758747.734109:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758747.734117:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758747.734128:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588655186304/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758747.734137:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758747.734141:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787758747.734142:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655186176:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787758747.734143:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758747.734150:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11804 00000100:00080000:2.0:1787758747.734154:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758747.734157:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758747.734160:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758747.734169:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588655186432/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758747.734183:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758747.734187:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758747.734189:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758747.734192:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11804 00000100:00080000:2.0:1787758747.734195:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758747.734197:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758747.734202:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758747.734210:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588655186560/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787758747.734214:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787758747.734223:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758747.734226:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758747.734228:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758747.734231:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11804 00000100:00080000:2.0:1787758747.734234:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758747.734237:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758747.734240:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758747.734248:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588655186688/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758747.734255:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758747.734258:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758747.734260:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787758747.734263:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758747/real 1787758747] req@ffff8c05b2e70e00 x1874588655186176/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758747.734266:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11804 00000100:00080000:2.0:1787758747.734271:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758747.734278:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758747.734282:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758747.734288:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588655186176/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758747.734295:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588655186816/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758747.734301:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758747.734303:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758747.734308:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787758747.734309:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787758747.734313:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787758747.734317:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655186176:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787758747.734321:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11804 00000100:00080000:2.0:1787758747.734327:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758747.734332:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758747.734338:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758747.734345:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655186304:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758747.734352:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588655186944/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758747.734371:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758747.734375:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758747.734377:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758747.734382:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11804 00000100:00080000:2.0:1787758747.734386:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787758747.734393:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787758747.734393:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758747.734402:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758747.734414:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758747/real 1787758747] req@ffff8c05b2e70700 x1874588655186304/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758747.734418:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588655187072/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758747.734449:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588655186304/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758747.734462:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758747.734467:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758747.734471:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655186304:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758747.734476:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655186432:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758747.734493:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758747.734502:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758747/real 1787758747] req@ffff8c05b2e73480 x1874588655186432/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758747.734521:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588655186432/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758747.734530:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758747.734546:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758747.734550:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655186432:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758747.734557:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655186560:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758747.734580:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758747.734589:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758747/real 1787758747] req@ffff8c05b2e71500 x1874588655186560/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758747.734598:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588655186560/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758747.734604:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758747.734606:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758747.734609:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655186560:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758747.734613:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655186688:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758747.734625:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758747.734631:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758747/real 1787758747] req@ffff8c05b2e73b80 x1874588655186688/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758747.734639:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588655186688/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758747.734649:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758747.734651:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758747.734652:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655186688:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758747.734656:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655186816:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758747.734665:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758747.734671:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758747/real 1787758747] req@ffff8c05b2e71c00 x1874588655186816/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758747.734677:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588655186816/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758747.734683:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758747.734685:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758747.734687:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655186816:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758747.734700:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655186944:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758747.734710:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758747.734716:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758747/real 1787758747] req@ffff8c05b2e71880 x1874588655186944/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758747.734723:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588655186944/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758747.734728:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758747.734731:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758747.734733:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655186944:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758747.734737:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655187072:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758747.734747:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758747.734752:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758747/real 1787758747] req@ffff8c05b2e70380 x1874588655187072/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758747.734759:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588655187072/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758747.734765:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758747.734766:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758747.734768:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655187072:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758747.862020:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758748.310030:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758748.885970:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758749.334108:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758749.910005:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758750.358095:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758750.933935:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758751.382072:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758751.958026:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758752.406086:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758752.853995:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758752.854006:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758752.854010:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758752.854020:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11809 00000100:00080000:1.0:1787758752.854030:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758752.854036:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758752.854089:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655187200/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758752.854114:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758752.854117:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758752.854119:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758752.854121:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11809 00000100:00080000:1.0:1787758752.854124:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758752.854127:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758752.854136:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758752.854147:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655187328/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758752.854154:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758752.854156:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758752.854157:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758752.854160:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11809 00000100:00080000:1.0:1787758752.854163:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758752.854166:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758752.854171:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758752.854174:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655187200:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758752.854186:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655187456/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758752.854199:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758752.854202:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758752.854204:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758752.854208:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11809 00000100:00080000:1.0:1787758752.854212:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758752.854216:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758752.854222:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758752.854234:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655187584/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787758752.854245:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758752.854245:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758752.854251:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758752.854252:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758752.854256:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11809 00000100:00080000:1.0:1787758752.854260:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758752.854263:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758752.854267:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758752.854277:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655187712/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758752.854282:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758752.854285:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758752.854286:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758752.854288:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11809 00000100:00080000:1.0:1787758752.854291:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758752.854293:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758752.854296:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758752.854300:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758752/real 1787758752] req@ffff8c05b2d99f80 x1874588655187200/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758752.854305:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655187840/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758752.854326:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758752.854330:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758752.854331:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787758752.854336:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655187200/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758752.854336:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11809 00000100:00080000:1.0:1787758752.854343:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758752.854350:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787758752.854357:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787758752.854357:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787758752.854365:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758752.854372:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655187968/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758752.854381:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655187200:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758752.854392:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758752.854396:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758752.854398:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787758752.854399:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655187328:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758752.854403:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11809 00000100:00080000:1.0:1787758752.854407:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758752.854410:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758752.854413:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758752.854422:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655188096/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787758752.854435:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758752.854464:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758752/real 1787758752] req@ffff8c05b2d98000 x1874588655187328/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758752.854477:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655187328/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758752.854486:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758752.854489:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758752.854492:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655187328:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758752.854517:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655187456:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758752.854534:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758752.854543:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758752/real 1787758752] req@ffff8c05b2d98700 x1874588655187456/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758752.854554:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655187456/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758752.854562:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758752.854565:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758752.854568:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655187456:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758752.854574:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655187584:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758752.854589:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758752.854598:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758752/real 1787758752] req@ffff8c05b2d9b480 x1874588655187584/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758752.854609:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655187584/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758752.854624:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758752.854627:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758752.854630:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655187584:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758752.854636:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655187712:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758752.854652:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758752.854662:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758752/real 1787758752] req@ffff8c05b2d9aa00 x1874588655187712/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758752.854673:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655187712/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758752.854686:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758752.854689:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758752.854693:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655187712:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758752.854698:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655187840:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758752.854713:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758752.854721:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758752/real 1787758752] req@ffff8c05b2d9bb80 x1874588655187840/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758752.854732:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655187840/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758752.854746:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758752.854757:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758752.854760:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655187840:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758752.854766:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655187968:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758752.854781:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758752.854790:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758752/real 1787758752] req@ffff8c05b2d99880 x1874588655187968/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758752.854800:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655187968/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758752.854808:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758752.854811:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758752.854814:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655187968:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758752.854820:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655188096:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758752.854834:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758752.854842:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758752/real 1787758752] req@ffff8c05b2d9b100 x1874588655188096/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758752.854852:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655188096/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758752.854867:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758752.854870:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758752.854897:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655188096:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758752.981976:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758753.430114:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758754.005946:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758754.454053:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758755.029956:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758755.478066:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758756.053983:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758756.502050:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758757.077944:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758757.526065:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758757.974002:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758757.974013:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787758757.974017:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758757.974028:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11814 00000100:00080000:2.0:1787758757.974040:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758757.974047:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758757.974089:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73800 x1874588655188224/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758757.974114:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758757.974117:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758757.974118:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758757.974121:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11814 00000100:00080000:2.0:1787758757.974123:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758757.974125:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758757.974133:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758757.974144:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588655188352/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758757.974152:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758757.974154:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758757.974157:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758757.974159:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11814 00000100:00080000:2.0:1787758757.974161:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758757.974166:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758757.974169:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758757.974174:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655188224:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787758757.974175:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588655188480/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758757.974189:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758757.974191:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758757.974192:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758757.974194:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11814 00000100:00080000:2.0:1787758757.974195:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758757.974197:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758757.974200:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758757.974207:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588655188608/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758757.974212:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758757.974215:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758757.974216:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758757.974218:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11814 00000100:00080000:2.0:1787758757.974221:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758757.974222:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758757.974225:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758757.974232:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588655188736/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758757.974238:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758757.974240:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758757.974241:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758757.974243:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11814 00000100:00080000:2.0:1787758757.974245:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758757.974246:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758757.974249:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787758757.974254:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758757.974257:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588655188864/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758757.974263:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758757.974266:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787758757.974267:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758757.974270:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11814 00000100:00080000:2.0:1787758757.974272:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758757.974274:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758757.974277:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758757.974284:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588655188992/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758757.974288:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758757.974290:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758757.974291:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758757.974294:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11814 00000100:00080000:2.0:1787758757.974295:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758757.974297:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758757.974300:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758757.974307:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588655189120/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758757.974312:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758757/real 1787758757] req@ffff8c05b2e73800 x1874588655188224/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758757.974334:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73800 x1874588655188224/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758757.974346:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758757.974350:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758757.974367:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655188224:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787758757.974392:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655188352:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758757.974414:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758757.974426:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758757/real 1787758757] req@ffff8c05b2e70380 x1874588655188352/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758757.974439:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588655188352/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758757.974448:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758757.974452:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758757.974455:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655188352:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758757.974462:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655188480:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758757.974481:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758757.974492:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758757/real 1787758757] req@ffff8c05b2e71880 x1874588655188480/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758757.974504:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588655188480/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758757.974511:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758757.974515:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758757.974518:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655188480:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758757.974524:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655188608:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758757.974541:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758757.974551:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758757/real 1787758757] req@ffff8c05b2e71c00 x1874588655188608/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758757.974562:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588655188608/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758757.974571:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758757.974574:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758757.974578:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655188608:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758757.974583:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655188736:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758757.974601:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758757.974610:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758757/real 1787758757] req@ffff8c05b2e73b80 x1874588655188736/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758757.974627:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588655188736/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758757.974635:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758757.974637:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758757.974640:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655188736:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758757.974646:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655188864:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758757.974670:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758757.974680:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758757/real 1787758757] req@ffff8c05b2e71500 x1874588655188864/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758757.974691:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588655188864/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758757.974700:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758757.974703:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758757.974706:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655188864:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758757.974712:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655188992:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758757.974727:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758757.974743:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758757/real 1787758757] req@ffff8c05b2e73480 x1874588655188992/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758757.974753:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588655188992/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758757.974767:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758757.974771:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758757.974774:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655188992:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758757.974779:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655189120:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758757.974797:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758757.974806:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758757/real 1787758757] req@ffff8c05b2e70700 x1874588655189120/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758757.974817:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588655189120/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758757.974826:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758757.974829:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758757.974832:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655189120:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758758.101951:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758758.550009:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758759.125971:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758759.574126:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758760.149978:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758760.598079:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758761.173991:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758761.622145:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758762.198017:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758762.646045:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758763.094003:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758763.094015:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758763.094020:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758763.094031:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11819 00000100:00080000:1.0:1787758763.094046:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758763.094053:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758763.094097:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655189248/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758763.094123:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758763.094126:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758763.094127:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758763.094131:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11819 00000100:00080000:1.0:1787758763.094133:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758763.094136:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758763.094145:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758763.094155:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588655189376/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758763.094163:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758763.094165:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758763.094167:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758763.094169:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11819 00000100:00080000:1.0:1787758763.094171:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758763.094173:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758763.094177:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758763.094181:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655189248:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758763.094187:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655189504/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758763.094194:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758763.094200:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758763.094202:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758763.094204:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11819 00000100:00080000:1.0:1787758763.094206:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758763.094209:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758763.094212:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758763.094222:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655189632/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758763.094228:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758763.094231:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758763.094232:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758763.094234:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11819 00000100:00080000:1.0:1787758763.094237:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758763.094239:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758763.094242:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758763.094249:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655189760/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758763.094255:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758763.094257:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758763.094259:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758763.094274:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11819 00000100:00080000:1.0:1787758763.094277:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787758763.094283:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758763.094284:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758763.094290:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758763.094300:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655189888/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758763.094307:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758763.094310:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758763.094311:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758763.094314:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11819 00000100:00080000:1.0:1787758763.094317:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758763.094320:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758763.094324:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758763.094340:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758763/real 1787758763] req@ffff8c05b2d98380 x1874588655189248/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758763.094340:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655190016/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758763.094368:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758763.094371:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758763.094373:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758763.094376:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11819 00000100:00080000:1.0:1787758763.094379:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787758763.094380:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655189248/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758763.094384:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758763.094390:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758763.094399:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655190144/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758763.094403:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758763.094412:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758763.094419:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655189248:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787758763.094429:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655189376:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758763.094453:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758763.094463:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758763/real 1787758763] req@ffff8c05b2d9a300 x1874588655189376/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758763.094473:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588655189376/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758763.094479:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758763.094481:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758763.094484:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655189376:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758763.094505:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655189504:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758763.094517:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758763.094524:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758763/real 1787758763] req@ffff8c05b2d9b100 x1874588655189504/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758763.094532:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655189504/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758763.094537:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758763.094539:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758763.094541:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655189504:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758763.094545:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655189632:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758763.094556:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758763.094562:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758763/real 1787758763] req@ffff8c05b2d99880 x1874588655189632/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758763.094570:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655189632/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758763.094575:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758763.094577:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758763.094579:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655189632:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758763.094582:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655189760:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758763.094592:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758763.094598:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758763/real 1787758763] req@ffff8c05b2d9bb80 x1874588655189760/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758763.094606:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655189760/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758763.094611:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758763.094613:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758763.094615:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655189760:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758763.094618:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655189888:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758763.094628:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758763.094634:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758763/real 1787758763] req@ffff8c05b2d9aa00 x1874588655189888/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758763.094642:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655189888/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758763.094647:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758763.094650:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758763.094652:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655189888:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758763.094655:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655190016:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758763.094664:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758763.094670:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758763/real 1787758763] req@ffff8c05b2d9b480 x1874588655190016/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758763.094688:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655190016/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758763.094697:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758763.094701:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758763.094705:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655190016:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758763.094711:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655190144:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758763.094733:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758763.094750:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758763/real 1787758763] req@ffff8c05b2d98700 x1874588655190144/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758763.094759:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655190144/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758763.094764:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758763.094767:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758763.094769:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655190144:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758763.222003:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758763.670109:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758764.245996:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758764.694027:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758765.270057:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758765.718126:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758766.293980:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758766.742088:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758767.318013:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758767.766149:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758768.214013:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758768.214024:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758768.214028:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758768.214039:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11824 00000100:00080000:1.0:1787758768.214051:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758768.214058:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758768.214102:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655190272/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758768.214128:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758768.214131:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758768.214132:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758768.214136:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11824 00000100:00080000:1.0:1787758768.214138:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758768.214140:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758768.214149:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758768.214158:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655190400/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758768.214165:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758768.214168:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758768.214169:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758768.214186:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11824 00000100:00080000:1.0:1787758768.214189:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758768.214192:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758768.214196:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758768.214207:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655190528/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758768.214214:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758768.214217:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758768.214219:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758768.214222:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11824 00000100:00080000:1.0:1787758768.214225:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758768.214228:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758768.214231:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758768.214239:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655190656/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758768.214246:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758768.214249:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758768.214251:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758768.214255:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11824 00000100:00080000:1.0:1787758768.214259:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787758768.214261:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655190272:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758768.214262:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758768.214267:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758768.214282:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655190784/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758768.214295:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758768.214297:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758768.214299:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758768.214303:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11824 00000100:00080000:1.0:1787758768.214305:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758768.214307:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758768.214310:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758768.214317:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655190912/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758768.214328:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758768.214330:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758768.214333:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758768.214337:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11824 00000100:00080000:1.0:1787758768.214349:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758768.214351:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758768.214354:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787758768.214364:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758768.214365:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655191040/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758768.214375:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758768.214378:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758768.214379:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758768.214383:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11824 00000100:00080000:1.0:1787758768.214386:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758768.214389:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758768.214392:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758768.214400:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655191168/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787758768.214433:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758768/real 1787758768] req@ffff8c05b2d98000 x1874588655190272/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758768.214456:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655190272/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758768.214468:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758768.214473:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758768.214478:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655190272:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787758768.214488:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655190400:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758768.214514:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758768.214527:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758768/real 1787758768] req@ffff8c05b2d99f80 x1874588655190400/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758768.214541:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655190400/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758768.214550:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758768.214554:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758768.214558:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655190400:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758768.214567:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655190528:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758768.214587:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758768.214598:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758768/real 1787758768] req@ffff8c05b2d98700 x1874588655190528/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758768.214623:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655190528/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758768.214632:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758768.214635:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758768.214639:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655190528:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758768.214669:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655190656:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758768.214691:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758768.214701:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758768/real 1787758768] req@ffff8c05b2d9b480 x1874588655190656/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758768.214712:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655190656/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758768.214720:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758768.214724:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758768.214727:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655190656:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758768.214735:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655190784:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758768.214752:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758768.214762:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758768/real 1787758768] req@ffff8c05b2d9aa00 x1874588655190784/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758768.214772:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655190784/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758768.214780:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758768.214783:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758768.214787:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655190784:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758768.214794:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655190912:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787758768.214814:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758768.214826:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758768/real 1787758768] req@ffff8c05b2d9bb80 x1874588655190912/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758768.214837:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655190912/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758768.214845:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758768.214848:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758768.214851:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655190912:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758768.214857:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655191040:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758768.214918:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758768.214931:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758768/real 1787758768] req@ffff8c05b2d99880 x1874588655191040/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758768.214944:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655191040/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758768.214960:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758768.214964:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758768.214967:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655191040:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758768.214975:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655191168:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787758768.214996:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758768.215008:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758768/real 1787758768] req@ffff8c05b2d9b100 x1874588655191168/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758768.215019:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655191168/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758768.215028:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758768.215031:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758768.215035:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655191168:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758768.341959:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758768.790009:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758769.365988:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758769.814140:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758770.389957:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758770.838121:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758771.413990:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758771.862168:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758772.437933:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758772.886077:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758773.333998:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758773.334011:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758773.334017:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758773.334030:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11829 00000100:00080000:0.0:1787758773.334043:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758773.334052:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758773.334114:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588655191296/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758773.334144:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758773.334149:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758773.334151:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758773.334156:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11829 00000100:00080000:0.0:1787758773.334161:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758773.334166:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758773.334176:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758773.334178:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655191296:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787758773.334195:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655191424/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758773.334208:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758773.334212:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758773.334215:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758773.334219:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11829 00000100:00080000:0.0:1787758773.334238:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758773.334242:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758773.334249:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787758773.334249:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758773.334264:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588655191552/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758773.334276:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758773.334281:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758773.334283:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758773.334287:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11829 00000100:00080000:0.0:1787758773.334292:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758773.334295:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758773/real 1787758773] req@ffff8c05a9f74e00 x1874588655191296/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758773.334300:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758773.334311:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758773.334318:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588655191296/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758773.334327:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588655191680/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758773.334330:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758773.334337:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758773.334343:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655191296:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758773.334348:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758773.334353:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758773.334355:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758773.334359:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11829 00000100:00080000:0.0:1787758773.334363:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758773.334367:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758773.334371:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655191424:192.168.201.107@tcp:38:kworker.0 00000008:00100000:0.0:1787758773.334374:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758773.334390:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588655191808/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787758773.334396:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758773.334409:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758773.334411:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758773/real 1787758773] req@ffff8c05a9f75880 x1874588655191424/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758773.334420:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758773.334424:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758773.334432:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11829 00000100:00100000:1.0:1787758773.334432:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655191424/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758773.334437:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758773.334444:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758773.334446:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:1.0:1787758773.334450:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758773.334454:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655191424:192.168.201.107@tcp:38:kworker.0 00000008:00100000:0.0:1787758773.334456:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758773.334460:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655191552:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758773.334474:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588655191936/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758773.334486:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787758773.334489:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758773.334492:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758773.334496:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758773.334501:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758773/real 1787758773] req@ffff8c05a9f75500 x1874588655191552/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758773.334503:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11829 00000100:00080000:0.0:1787758773.334511:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758773.334517:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758773.334520:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588655191552/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787758773.334525:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787758773.334536:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758773.334540:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758773.334544:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588655192064/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758773.334545:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655191552:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758773.334556:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655191680:192.168.201.107@tcp:8:kworker.0 00000100:00080000:0.0:1787758773.334567:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758773.334571:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758773.334574:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787758773.334579:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758773.334583:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11829 00000100:00080000:0.0:1787758773.334588:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758773.334591:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758773/real 1787758773] req@ffff8c05a9f74000 x1874588655191680/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758773.334596:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758773.334607:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758773.334609:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588655191680/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758773.334619:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787758773.334624:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588655192192/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758773.334625:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758773.334631:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655191680:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758773.334650:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655191808:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758773.334672:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758773.334680:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758773/real 1787758773] req@ffff8c05a9f75180 x1874588655191808/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758773.334687:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588655191808/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758773.334693:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758773.334696:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758773.334700:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655191808:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758773.334706:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655191936:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758773.334724:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758773.334736:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758773/real 1787758773] req@ffff8c05a9f76d80 x1874588655191936/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758773.334750:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588655191936/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758773.334757:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758773.334759:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758773.334761:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655191936:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758773.334766:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655192064:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758773.334780:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758773.334788:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758773/real 1787758773] req@ffff8c05a9f77100 x1874588655192064/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758773.334795:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588655192064/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758773.334801:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758773.334802:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758773.334804:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655192064:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758773.334808:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655192192:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758773.334817:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758773.334824:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758773/real 1787758773] req@ffff8c05a9f74e00 x1874588655192192/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758773.334831:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588655192192/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758773.334836:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758773.334838:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758773.334839:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655192192:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758773.461987:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758773.910130:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758774.485969:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758774.934091:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758775.509979:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758775.958172:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758776.534009:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758776.982070:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758777.557970:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758778.006120:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758778.454023:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758778.454037:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758778.454044:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758778.454058:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11834 00000100:00080000:0.0:1787758778.454071:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758778.454080:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758778.454128:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655192320/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758778.454170:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758778.454174:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758778.454176:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758778.454180:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11834 00000100:00080000:0.0:1787758778.454185:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758778.454191:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758778.454203:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758778.454221:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655192448/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758778.454227:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655192320:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758778.454241:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758778.454247:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758778.454249:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758778.454255:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11834 00000100:00080000:0.0:1787758778.454260:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758778.454265:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758778.454272:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758778.454288:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655192576/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758778.454301:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787758778.454306:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758778.454309:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758778.454314:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758778.454321:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11834 00000100:00080000:0.0:1787758778.454335:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758778.454341:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758778.454347:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758778.454351:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758778/real 1787758778] req@ffff8c05a9f74a80 x1874588655192320/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758778.454365:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588655192704/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758778.454372:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655192320/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758778.454384:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758778.454388:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758778.454390:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758778.454394:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758778.454399:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787758778.454404:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11834 00000100:00080000:0.0:1787758778.454410:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787758778.454410:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655192320:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758778.454419:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758778.454426:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758778.454441:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588655192832/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758778.454445:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655192448:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758778.454462:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758778.454466:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758778.454468:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787758778.454475:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758778.454482:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11834 00000100:00080000:0.0:1787758778.454489:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787758778.454489:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758778/real 1787758778] req@ffff8c05a9f77800 x1874588655192448/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758778.454497:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758778.454507:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758778.454507:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655192448/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758778.454518:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758778.454521:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758778.454526:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655192448:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758778.454528:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588655192960/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758778.454533:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655192576:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758778.454550:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758778.454554:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758778.454557:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787758778.454558:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758778.454567:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11834 00000100:00100000:2.0:1787758778.454573:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758778/real 1787758778] req@ffff8c05a9f74380 x1874588655192576/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758778.454574:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758778.454585:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758778.454590:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655192576/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787758778.454595:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787758778.454608:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758778.454611:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758778.454618:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655192576:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758778.454620:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655193088/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758778.454626:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655192704:192.168.201.107@tcp:8:kworker.0 00000100:00080000:0.0:1787758778.454643:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758778.454648:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758778.454651:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758778.454658:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11834 00000800:00000100:2.0:1787758778.454664:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758778.454667:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758778.454674:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758778.454681:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758778.454687:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758778/real 1787758778] req@ffff8c05a9f75c00 x1874588655192704/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758778.454696:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655193216/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758778.454702:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588655192704/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758778.454723:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758778.454727:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758778.454731:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655192704:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758778.454746:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655192832:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758778.454762:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758778.454769:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758778/real 1787758778] req@ffff8c05a9f76a00 x1874588655192832/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758778.454777:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588655192832/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758778.454783:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758778.454785:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758778.454787:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655192832:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758778.454791:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655192960:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758778.454801:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758778.454807:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758778/real 1787758778] req@ffff8c05a9f77b80 x1874588655192960/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758778.454815:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588655192960/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758778.454820:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758778.454822:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758778.454824:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655192960:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758778.454827:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655193088:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758778.454836:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758778.454842:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758778/real 1787758778] req@ffff8c05a9f76300 x1874588655193088/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758778.454849:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655193088/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758778.454855:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758778.454856:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758778.454858:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655193088:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758778.454861:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655193216:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758778.454907:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758778.454918:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758778/real 1787758778] req@ffff8c05a9f76680 x1874588655193216/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758778.454928:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655193216/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758778.454934:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758778.454936:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758778.454939:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655193216:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758778.581944:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758779.030056:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758779.605969:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758780.054149:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758780.629992:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758781.078032:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758781.653991:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758782.102208:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758782.678004:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758783.126106:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758783.574009:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758783.574019:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758783.574024:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758783.574035:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11839 00000100:00080000:0.0:1787758783.574046:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758783.574053:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758783.574096:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655193344/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758783.574127:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758783.574131:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758783.574132:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758783.574136:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11839 00000100:00080000:0.0:1787758783.574138:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758783.574141:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758783.574149:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758783.574158:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655193472/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758783.574166:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758783.574170:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758783.574171:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758783.574174:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11839 00000100:00080000:0.0:1787758783.574175:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758783.574178:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758783.574182:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758783.574189:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588655193600/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758783.574194:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655193344:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758783.574210:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758783.574212:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758783.574214:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758783.574217:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11839 00000100:00080000:0.0:1787758783.574220:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758783.574222:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758783.574232:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758783.574245:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588655193728/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758783.574252:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758783.574255:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758783.574256:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758783.574259:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11839 00000100:00080000:0.0:1787758783.574262:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758783.574265:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758783.574269:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787758783.574274:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758783.574278:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588655193856/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758783.574286:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758783.574289:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758783.574290:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758783.574293:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11839 00000100:00080000:0.0:1787758783.574299:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758783.574301:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758783.574304:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758783.574313:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655193984/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758783.574320:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787758783.574322:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758783/real 1787758783] req@ffff8c05a9f76680 x1874588655193344/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758783.574323:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758783.574327:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758783.574334:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11839 00000100:00080000:0.0:1787758783.574338:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758783.574345:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758783.574348:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758783.574353:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655193344/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758783.574358:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655194112/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758783.574376:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758783.574376:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758783.574383:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758783.574386:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758783.574386:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787758783.574395:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11839 00000100:00100000:2.0:1787758783.574395:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655193344:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758783.574397:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758783.574404:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758783.574408:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655193472:192.168.201.107@tcp:38:kworker.0 00000008:00100000:0.0:1787758783.574411:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758783.574422:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655194240/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787758783.574433:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758783.574449:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758783/real 1787758783] req@ffff8c05a9f76300 x1874588655193472/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758783.574460:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655193472/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758783.574468:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758783.574471:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758783.574474:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655193472:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758783.574494:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655193600:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758783.574506:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758783.574513:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758783/real 1787758783] req@ffff8c05a9f77b80 x1874588655193600/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758783.574520:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588655193600/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758783.574525:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758783.574527:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758783.574530:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655193600:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758783.574536:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655193728:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758783.574555:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758783.574566:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758783/real 1787758783] req@ffff8c05a9f76a00 x1874588655193728/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758783.574580:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588655193728/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758783.574588:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758783.574591:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758783.574593:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655193728:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758783.574597:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655193856:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758783.574612:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758783.574619:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758783/real 1787758783] req@ffff8c05a9f75c00 x1874588655193856/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758783.574627:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588655193856/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758783.574632:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758783.574634:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758783.574637:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655193856:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758783.574640:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655193984:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758783.574649:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758783.574655:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758783/real 1787758783] req@ffff8c05a9f74380 x1874588655193984/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758783.574662:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655193984/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758783.574666:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758783.574668:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758783.574671:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655193984:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758783.574674:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655194112:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758783.574685:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758783.574691:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758783/real 1787758783] req@ffff8c05a9f77800 x1874588655194112/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758783.574698:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655194112/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758783.574703:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758783.574704:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758783.574706:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655194112:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758783.574709:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655194240:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758783.574719:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758783.574724:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758783/real 1787758783] req@ffff8c05a9f74a80 x1874588655194240/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758783.574731:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655194240/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758783.574736:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758783.574738:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758783.574739:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655194240:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758783.701985:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758784.150122:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758784.726037:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758785.174126:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758785.749992:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758786.198157:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758786.774011:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758787.222109:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758787.797951:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758788.246085:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758788.694025:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758788.694039:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787758788.694045:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758788.694058:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11845 00000100:00080000:2.0:1787758788.694072:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758788.694080:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758788.694128:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588655194368/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758788.694166:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758788.694170:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758788.694172:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758788.694177:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11845 00000100:00080000:2.0:1787758788.694181:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758788.694186:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758788.694197:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758788.694213:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588655194496/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758788.694218:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655194368:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787758788.694235:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758788.694239:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758788.694241:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758788.694247:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11845 00000100:00080000:2.0:1787758788.694251:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758788.694256:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758788.694262:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758788.694275:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588655194624/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758788.694287:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787758788.694294:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787758788.694295:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758788.694300:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758788.694305:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11845 00000100:00080000:2.0:1787758788.694310:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758788.694315:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758788.694321:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758788.694335:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588655194752/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758788.694343:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758788/real 1787758788] req@ffff8c05b2e70e00 x1874588655194368/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758788.694350:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758788.694356:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758788.694359:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758788.694364:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588655194368/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758788.694367:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11845 00000100:00080000:2.0:1787758788.694376:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758788.694380:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758788.694384:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:1.0:1787758788.694388:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758788.694394:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655194368:192.168.201.107@tcp:250:kworker.0 00000008:00100000:2.0:1787758788.694395:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758788.694412:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588655194880/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758788.694419:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655194496:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787758788.694432:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758788.694438:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758788.694441:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787758788.694458:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787758788.694458:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11845 00000100:00080000:2.0:1787758788.694470:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758788.694476:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758788.694480:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758788/real 1787758788] req@ffff8c05b2e71180 x1874588655194496/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787758788.694483:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758788.694505:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588655194496/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758788.694506:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588655195008/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758788.694528:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758788.694533:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758788.694536:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787758788.694541:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787758788.694546:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655194496:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787758788.694546:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758788.694556:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11845 00000100:00080000:2.0:1787758788.694560:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758788.694561:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655194624:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787758788.694567:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758788.694574:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758788.694589:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588655195136/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787758788.694599:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787758788.694608:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758788.694612:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758788.694615:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758788.694620:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758788/real 1787758788] req@ffff8c05b2e72680 x1874588655194624/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758788.694623:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11845 00000100:00080000:2.0:1787758788.694631:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758788.694638:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758788.694646:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588655194624/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787758788.694646:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787758788.694662:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758788.694667:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758788.694671:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655194624:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758788.694679:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655194752:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758788.694694:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588655195264/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787758788.694704:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758788.694723:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758788/real 1787758788] req@ffff8c05b2e72a00 x1874588655194752/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758788.694742:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588655194752/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758788.694749:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758788.694752:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758788.694754:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655194752:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758788.694765:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655194880:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758788.694777:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758788.694784:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758788/real 1787758788] req@ffff8c05b2e72300 x1874588655194880/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758788.694793:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588655194880/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758788.694798:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758788.694800:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758788.694803:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655194880:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758788.694825:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655195008:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758788.694840:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758788.694848:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758788/real 1787758788] req@ffff8c05b2e70a80 x1874588655195008/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758788.694855:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588655195008/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758788.694865:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758788.694867:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758788.694869:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655195008:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758788.694873:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655195136:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758788.694938:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758788.694953:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758788/real 1787758788] req@ffff8c05b2e70000 x1874588655195136/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758788.694961:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588655195136/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758788.694967:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758788.694969:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758788.694971:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655195136:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758788.694975:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655195264:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758788.694988:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758788.694994:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758788/real 1787758788] req@ffff8c05b2e73100 x1874588655195264/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758788.695002:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588655195264/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758788.695007:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758788.695009:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758788.695011:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655195264:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758788.822038:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758789.270055:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758789.846007:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758790.294123:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758790.870061:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758791.318098:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758791.894111:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758792.342108:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758792.918012:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758793.366085:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758793.814045:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758793.814054:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758793.814057:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758793.814065:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11850 00000100:00080000:1.0:1787758793.814075:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758793.814082:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758793.814116:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588655195392/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758793.814143:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758793.814146:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758793.814148:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758793.814151:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11850 00000100:00080000:1.0:1787758793.814153:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758793.814156:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758793.814164:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758793.814173:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655195520/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758793.814183:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758793.814186:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758793.814187:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758793.814189:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11850 00000100:00080000:1.0:1787758793.814191:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758793.814196:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758793.814201:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758793.814217:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655195392:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758793.814218:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655195648/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758793.814242:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758793.814247:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758793.814250:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758793.814257:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11850 00000100:00080000:1.0:1787758793.814263:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758793.814266:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758793.814274:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758793.814284:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655195776/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758793.814291:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758793.814294:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758793.814296:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758793.814299:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11850 00000800:00000100:2.0:1787758793.814308:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758793.814311:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758793.814317:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758793.814322:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758793.814340:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655195904/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758793.814348:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758793.814351:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758793.814353:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758793.814357:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11850 00000100:00080000:1.0:1787758793.814360:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758793.814363:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758793.814371:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758793.814371:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758793/real 1787758793] req@ffff8c05b2d9a300 x1874588655195392/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758793.814388:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655196032/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758793.814396:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758793.814399:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758793.814405:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787758793.814406:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588655195392/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758793.814412:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11850 00000100:00080000:1.0:1787758793.814416:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758793.814424:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758793.814436:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787758793.814451:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787758793.814452:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655196160/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758793.814469:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787758793.814471:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758793.814478:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758793.814482:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787758793.814482:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655195392:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758793.814492:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11850 00000100:00080000:1.0:1787758793.814501:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758793.814504:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758793.814511:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655195520:192.168.201.107@tcp:38:kworker.0 00000008:00100000:1.0:1787758793.814512:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758793.814537:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655196288/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787758793.814556:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758793.814577:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758793/real 1787758793] req@ffff8c05b2d98380 x1874588655195520/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758793.814594:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655195520/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758793.814612:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758793.814616:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758793.814620:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655195520:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758793.814647:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655195648:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758793.814665:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758793.814673:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758793/real 1787758793] req@ffff8c05b2d9b100 x1874588655195648/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758793.814681:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655195648/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758793.814687:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758793.814690:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758793.814692:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655195648:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758793.814696:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655195776:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758793.814708:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758793.814714:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758793/real 1787758793] req@ffff8c05b2d99880 x1874588655195776/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758793.814737:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655195776/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758793.814743:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758793.814745:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758793.814748:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655195776:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758793.814753:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655195904:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758793.814766:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758793.814774:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758793/real 1787758793] req@ffff8c05b2d9bb80 x1874588655195904/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758793.814782:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655195904/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758793.814787:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758793.814789:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758793.814791:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655195904:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758793.814794:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655196032:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758793.814805:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758793.814812:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758793/real 1787758793] req@ffff8c05b2d9aa00 x1874588655196032/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758793.814819:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655196032/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758793.814824:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758793.814826:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758793.814828:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655196032:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758793.814831:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655196160:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758793.814843:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758793.814850:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758793/real 1787758793] req@ffff8c05b2d9b480 x1874588655196160/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758793.814857:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655196160/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758793.814862:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758793.814864:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758793.814866:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655196160:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758793.814869:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655196288:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758793.814902:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758793.814910:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758793/real 1787758793] req@ffff8c05b2d98700 x1874588655196288/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758793.814918:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655196288/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758793.814924:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758793.814926:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758793.814929:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655196288:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758793.942034:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758794.390154:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758794.965989:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758795.414128:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758795.989996:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758796.438124:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758797.013998:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758797.462151:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758798.037958:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758798.486177:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758798.934004:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758798.934016:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758798.934021:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758798.934033:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11855 00000100:00080000:1.0:1787758798.934044:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758798.934052:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758798.934102:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655196416/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758798.934130:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758798.934133:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758798.934134:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758798.934138:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11855 00000100:00080000:1.0:1787758798.934141:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758798.934144:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758798.934156:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758798.934171:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655196544/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758798.934179:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758798.934184:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787758798.934184:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655196416:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758798.934188:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758798.934196:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11855 00000100:00080000:1.0:1787758798.934198:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758798.934201:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758798.934204:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758798.934214:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655196672/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758798.934221:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758798.934224:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758798.934225:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758798.934228:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11855 00000100:00080000:1.0:1787758798.934230:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758798.934233:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758798.934236:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758798.934244:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655196800/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758798.934250:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758798.934252:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758798.934254:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758798.934256:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11855 00000100:00080000:1.0:1787758798.934259:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758798.934261:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758798.934264:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758798.934281:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655196928/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758798.934291:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758798.934294:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758798.934296:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758798.934302:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11855 00000100:00080000:1.0:1787758798.934306:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787758798.934312:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655196544:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758798.934313:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758798.934324:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758798.934340:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655197056/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758798.934351:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758798.934353:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758798.934355:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787758798.934357:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655196672:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758798.934361:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11855 00000100:00080000:1.0:1787758798.934367:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758798.934371:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758798.934375:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758798.934387:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655196800:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758798.934389:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655197184/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758798.934402:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758798.934405:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758798.934406:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758798.934410:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11855 00000100:00080000:1.0:1787758798.934413:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758798.934415:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758798.934419:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655196928:192.168.201.107@tcp:38:kworker.0 00000008:00100000:1.0:1787758798.934422:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758798.934434:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655197312/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758798.934457:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655197056:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758798.934475:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655197184:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758798.934488:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655197312:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758799.061984:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758799.510025:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758800.085987:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758800.534184:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758801.110000:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758801.558086:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787758802.006166:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.107:988 00000400:00000100:3.0:1787758802.006214:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.107@tcp at host 192.168.201.107 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787758802.006230:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:3.0:1787758802.006271:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:3.0:1787758802.006279:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:3.0:1787758802.006286:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:3.0:1787758802.006292:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:3.0:1787758802.006300:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:3.0:1787758802.006317:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:3.0:1787758802.006330:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:2.0:1787758802.006377:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758798/real 1787758802] req@ffff8c05b2d9bb80 x1874588655197056/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758802.006420:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655197056/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758802.006435:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758802.006441:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758802.006446:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655197056:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758802.006456:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758798/real 1787758802] req@ffff8c05b2d99880 x1874588655197184/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758802.006470:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655197184/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758802.006484:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758802.006487:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758802.006491:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655197184:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758802.006499:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758798/real 1787758802] req@ffff8c05b2d9b100 x1874588655197312/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758802.006510:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655197312/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758802.006519:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758802.006523:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758802.006526:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655197312:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758802.006535:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758798/real 1787758802] req@ffff8c05b2d98000 x1874588655196544/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758802.006547:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655196544/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758802.006557:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758802.006561:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758802.006564:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655196544:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758802.006573:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758798/real 1787758802] req@ffff8c05b2d98700 x1874588655196672/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758802.006606:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655196672/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758802.006616:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758802.006621:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758802.006625:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655196672:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758802.006632:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758798/real 1787758802] req@ffff8c05b2d9b480 x1874588655196800/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758802.006646:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655196800/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758802.006655:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758802.006659:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758802.006663:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655196800:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758802.006670:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758798/real 1787758802] req@ffff8c05b2d9aa00 x1874588655196928/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758802.006682:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655196928/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758802.006691:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758802.006695:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758802.006699:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655196928:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758802.006707:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758798/real 1787758802] req@ffff8c05b2d99f80 x1874588655196416/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758802.006723:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655196416/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758802.006733:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758802.006736:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758802.006739:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655196416:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787758802.134030:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758802.582148:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758803.158007:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758803.606034:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758804.054006:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758804.054015:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787758804.054018:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758804.054026:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11860 00000100:00080000:2.0:1787758804.054036:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758804.054043:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758804.054079:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588655197440/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758804.054106:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758804.054108:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758804.054110:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758804.054113:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11860 00000100:00080000:2.0:1787758804.054115:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758804.054119:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758804.054127:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758804.054142:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588655197568/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758804.054143:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655197440:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787758804.054160:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758804.054165:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758804.054176:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758804.054182:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11860 00000100:00080000:2.0:1787758804.054187:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758804.054191:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758804.054197:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758804.054217:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588655197696/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787758804.054224:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787758804.054235:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758804.054238:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758804.054240:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758804.054245:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11860 00000100:00080000:2.0:1787758804.054249:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758804.054252:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758804.054256:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758804.054265:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588655197824/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758804.054273:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758804.054277:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1787758804.054278:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758804/real 1787758804] req@ffff8c05b2e72d80 x1874588655197440/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758804.054279:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758804.054285:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11860 00000100:00080000:2.0:1787758804.054289:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758804.054295:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758804.054303:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758804.054310:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588655197952/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758804.054315:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758804.054317:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758804.054319:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758804.054323:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11860 00000100:00080000:2.0:1787758804.054327:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758804.054328:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588655197440/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758804.054342:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758804.054350:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787758804.054351:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758804.054359:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758804.054365:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588655198080/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758804.054367:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655197440:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787758804.054385:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758804.054389:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787758804.054391:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758804.054397:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11860 00000100:00080000:2.0:1787758804.054401:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758804.054408:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655197568:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787758804.054409:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758804.054418:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758804.054434:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588655198208/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758804.054445:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787758804.054450:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787758804.054451:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758804.054456:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758804.054463:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11860 00000100:00080000:2.0:1787758804.054468:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758804.054474:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758804/real 1787758804] req@ffff8c05b2e73100 x1874588655197568/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758804.054475:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758804.054494:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588655197568/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787758804.054496:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758804.054515:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588655198336/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758804.054518:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758804.054529:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758804.054537:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655197568:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758804.054569:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655197696:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758804.054597:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758804.054609:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758804/real 1787758804] req@ffff8c05b2e70000 x1874588655197696/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758804.054623:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588655197696/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758804.054632:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758804.054637:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758804.054640:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655197696:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758804.054647:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655197824:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758804.054671:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758804.054683:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758804/real 1787758804] req@ffff8c05b2e70a80 x1874588655197824/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758804.054696:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588655197824/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758804.054705:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758804.054708:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758804.054713:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655197824:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758804.054719:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655197952:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758804.054739:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758804.054750:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758804/real 1787758804] req@ffff8c05b2e72300 x1874588655197952/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758804.054762:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588655197952/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758804.054771:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758804.054775:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758804.054779:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655197952:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758804.054785:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655198080:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758804.054806:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758804.054818:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758804/real 1787758804] req@ffff8c05b2e72a00 x1874588655198080/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758804.054829:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588655198080/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758804.054837:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758804.054841:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758804.054843:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655198080:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758804.054864:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655198208:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758804.054923:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758804.054936:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758804/real 1787758804] req@ffff8c05b2e72680 x1874588655198208/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758804.054950:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588655198208/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758804.054959:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758804.054962:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758804.054966:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655198208:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758804.054972:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655198336:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758804.055005:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758804.055018:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758804/real 1787758804] req@ffff8c05b2e71180 x1874588655198336/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758804.055029:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588655198336/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758804.055039:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758804.055042:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758804.055045:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655198336:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758804.182054:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758804.630154:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758805.205979:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758805.654036:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758806.230011:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758806.678080:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758807.253961:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758807.702017:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758808.277955:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758808.726152:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758809.174015:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758809.174026:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758809.174030:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758809.174041:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11865 00000100:00080000:0.0:1787758809.174052:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758809.174059:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758809.174103:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588655198464/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758809.174147:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758809.174151:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758809.174153:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758809.174158:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11865 00000100:00080000:0.0:1787758809.174162:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758809.174166:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758809.174175:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758809.174188:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588655198592/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758809.174198:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758809.174202:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787758809.174203:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655198464:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758809.174204:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758809.174212:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11865 00000100:00080000:0.0:1787758809.174215:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758809.174218:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758809.174221:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758809.174230:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588655198720/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758809.174236:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758809.174240:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758809.174241:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758809.174244:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11865 00000100:00080000:0.0:1787758809.174246:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758809.174248:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758809.174251:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758809.174260:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588655198848/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758809.174266:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758809.174268:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758809.174270:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758809.174274:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11865 00000100:00080000:0.0:1787758809.174277:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758809.174279:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758809.174283:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787758809.174287:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758809.174292:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588655198976/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758809.174300:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758809.174302:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758809.174304:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758809.174307:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11865 00000100:00080000:0.0:1787758809.174311:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758809.174314:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758809.174317:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758809.174324:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588655199104/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758809.174330:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758809.174333:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758809.174337:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758809.174337:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758809/real 1787758809] req@ffff8c05a9f74e00 x1874588655198464/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758809.174346:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11865 00000100:00080000:0.0:1787758809.174351:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758809.174357:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758809.174364:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758809.174375:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655199232/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758809.174379:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588655198464/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758809.174397:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758809.174401:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758809.174401:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758809.174403:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758809.174409:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787758809.174410:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11865 00000100:00080000:0.0:1787758809.174416:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758809.174416:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655198464:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758809.174423:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758809.174428:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758809.174436:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655199360/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758809.174446:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655198592:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758809.174485:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758809.174495:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758809/real 1787758809] req@ffff8c05a9f77100 x1874588655198592/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758809.174504:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588655198592/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758809.174510:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758809.174512:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758809.174514:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655198592:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758809.174518:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655198720:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758809.174529:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758809.174536:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758809/real 1787758809] req@ffff8c05a9f76d80 x1874588655198720/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758809.174543:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588655198720/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758809.174552:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758809.174554:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758809.174556:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655198720:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758809.174559:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655198848:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758809.174570:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758809.174575:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758809/real 1787758809] req@ffff8c05a9f75180 x1874588655198848/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758809.174582:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588655198848/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758809.174587:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758809.174589:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758809.174591:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655198848:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758809.174594:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655198976:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758809.174604:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758809.174610:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758809/real 1787758809] req@ffff8c05a9f74000 x1874588655198976/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758809.174617:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588655198976/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758809.174622:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758809.174624:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758809.174626:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655198976:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758809.174630:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655199104:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758809.174639:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758809.174644:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758809/real 1787758809] req@ffff8c05a9f75500 x1874588655199104/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758809.174651:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588655199104/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758809.174656:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758809.174658:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758809.174660:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655199104:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758809.174663:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655199232:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758809.174672:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758809.174678:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758809/real 1787758809] req@ffff8c05a9f75880 x1874588655199232/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758809.174685:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655199232/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758809.174690:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758809.174692:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758809.174694:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655199232:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758809.174707:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655199360:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758809.174717:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758809.174723:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758809/real 1787758809] req@ffff8c05a9f74a80 x1874588655199360/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758809.174730:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655199360/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758809.174735:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758809.174737:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758809.174739:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655199360:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758809.302001:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758809.750082:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758810.325974:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758810.774126:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758811.349955:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758811.798038:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758812.374019:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758812.822142:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758813.397996:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758813.846041:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758814.293977:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758814.293989:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758814.293994:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758814.294006:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11870 00000100:00080000:0.0:1787758814.294017:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758814.294026:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758814.294073:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655199488/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758814.294113:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758814.294117:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758814.294122:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758814.294127:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11870 00000100:00080000:0.0:1787758814.294132:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758814.294145:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758814.294158:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758814.294177:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655199616/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758814.294187:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758814.294191:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758814.294194:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758814.294200:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11870 00000100:00080000:0.0:1787758814.294204:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758814.294211:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758814.294218:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655199488:192.168.201.107@tcp:250:kworker.0 00000008:00100000:0.0:1787758814.294222:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758814.294251:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588655199744/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758814.294267:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758814.294271:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758814.294273:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758814.294279:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11870 00000100:00080000:0.0:1787758814.294285:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758814.294290:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758814.294297:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758814.294314:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588655199872/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787758814.294321:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758814.294335:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758814.294341:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758814.294343:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758814.294350:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11870 00000100:00080000:0.0:1787758814.294355:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758814.294374:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758814.294383:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758814.294399:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758814/real 1787758814] req@ffff8c05a9f77800 x1874588655199488/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758814.294402:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588655200000/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758814.294446:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758814.294447:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655199488/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758814.294466:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758814.294474:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758814.294478:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758814.294485:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11870 00000100:00080000:1.0:1787758814.294486:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758814.294496:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655199488:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758814.294505:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655199616:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758814.294507:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758814.294517:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758814.294525:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758814.294544:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655200128/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787758814.294549:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758814.294568:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758814.294574:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758814.294578:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758814.294584:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11870 00000100:00080000:0.0:1787758814.294589:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787758814.294590:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758814/real 1787758814] req@ffff8c05a9f74380 x1874588655199616/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758814.294606:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758814.294626:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758814.294642:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655199616/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758814.294650:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655200256/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758814.294659:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758814.294666:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758814.294667:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758814.294681:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655199616:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758814.294682:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758814.294692:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758814.294701:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11870 00000100:00080000:0.0:1787758814.294707:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758814.294713:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758814.294719:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758814.294725:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655199744:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758814.294740:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655200384/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787758814.294749:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758814.294772:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758814/real 1787758814] req@ffff8c05a9f75c00 x1874588655199744/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758814.294790:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588655199744/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758814.294800:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758814.294804:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758814.294808:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655199744:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758814.294817:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655199872:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758814.294837:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758814.294846:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758814/real 1787758814] req@ffff8c05a9f76a00 x1874588655199872/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758814.294854:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588655199872/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758814.294865:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758814.294867:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758814.294869:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655199872:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758814.294920:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655200000:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758814.294944:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758814.294954:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758814/real 1787758814] req@ffff8c05a9f77b80 x1874588655200000/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758814.294962:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588655200000/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758814.294968:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758814.294970:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758814.294972:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655200000:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758814.294977:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655200128:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758814.294988:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758814.294995:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758814/real 1787758814] req@ffff8c05a9f76300 x1874588655200128/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758814.295002:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655200128/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758814.295007:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758814.295009:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758814.295012:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655200128:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758814.295016:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655200256:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758814.295026:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758814.295032:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758814/real 1787758814] req@ffff8c05a9f76680 x1874588655200256/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758814.295039:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655200256/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758814.295044:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758814.295046:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758814.295048:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655200256:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758814.295062:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655200384:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758814.295080:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758814.295091:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758814/real 1787758814] req@ffff8c05a9f74380 x1874588655200384/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758814.295105:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655200384/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758814.295115:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758814.295118:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758814.295122:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655200384:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758814.422007:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758814.870123:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758815.445951:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758815.894093:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758816.470007:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758816.918119:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758817.493941:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758817.942187:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758818.517985:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758818.966142:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758819.414034:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758819.414045:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758819.414050:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758819.414061:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11875 00000100:00080000:1.0:1787758819.414073:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758819.414081:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758819.414136:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655200512/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758819.414180:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758819.414184:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758819.414186:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758819.414190:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11875 00000100:00080000:1.0:1787758819.414194:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758819.414199:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758819.414209:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758819.414226:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588655200640/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758819.414238:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758819.414242:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758819.414244:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758819.414250:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11875 00000100:00100000:2.0:1787758819.414252:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655200512:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758819.414254:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758819.414262:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758819.414282:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758819.414302:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655200768/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758819.414315:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758819.414319:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758819.414321:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758819.414326:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11875 00000100:00080000:1.0:1787758819.414331:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758819.414334:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758819.414339:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787758819.414347:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758819.414353:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655200896/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758819.414365:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758819.414370:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758819.414372:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758819.414377:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11875 00000100:00080000:1.0:1787758819.414382:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758819.414387:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758819.414393:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758819/real 1787758819] req@ffff8c05b2d98380 x1874588655200512/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787758819.414394:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758819.414410:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655201024/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758819.414421:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655200512/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758819.414424:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758819.414434:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758819.414438:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758819.414444:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758819.414449:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758819.414454:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655200512:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758819.414457:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11875 00000100:00080000:1.0:1787758819.414465:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758819.414471:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758819.414471:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655200640:192.168.201.107@tcp:38:kworker.0 00000008:00100000:1.0:1787758819.414480:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758819.414496:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655201152/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758819.414507:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758819.414512:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758819.414514:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787758819.414518:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758819.414520:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11875 00000100:00080000:1.0:1787758819.414527:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758819.414532:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758819.414538:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758819.414539:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758819/real 1787758819] req@ffff8c05b2d9a300 x1874588655200640/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758819.414553:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655201280/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758819.414558:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588655200640/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758819.414576:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758819.414581:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758819.414582:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758819.414587:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787758819.414588:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787758819.414593:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655200640:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758819.414597:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11875 00000100:00080000:1.0:1787758819.414604:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758819.414609:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758819.414615:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758819.414620:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655200768:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758819.414630:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655201408/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787758819.414651:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758819.414662:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758819/real 1787758819] req@ffff8c05b2d99f80 x1874588655200768/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758819.414671:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655200768/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758819.414680:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758819.414684:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758819.414688:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655200768:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758819.414696:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655200896:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758819.414717:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758819.414729:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758819/real 1787758819] req@ffff8c05b2d9aa00 x1874588655200896/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758819.414739:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655200896/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758819.414745:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758819.414748:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758819.414750:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655200896:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758819.414754:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655201024:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758819.414769:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758819.414775:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758819/real 1787758819] req@ffff8c05b2d9b480 x1874588655201024/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758819.414783:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655201024/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758819.414789:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758819.414790:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758819.414792:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655201024:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758819.414796:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655201152:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758819.414806:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758819.414812:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758819/real 1787758819] req@ffff8c05b2d98700 x1874588655201152/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758819.414820:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655201152/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758819.414825:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758819.414827:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758819.414829:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655201152:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758819.414832:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655201280:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758819.414841:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758819.414847:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758819/real 1787758819] req@ffff8c05b2d98000 x1874588655201280/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758819.414854:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655201280/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758819.414859:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758819.414861:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758819.414863:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655201280:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758819.414915:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655201408:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758819.414938:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758819.414949:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758819/real 1787758819] req@ffff8c05b2d9b100 x1874588655201408/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758819.414957:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655201408/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758819.414967:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758819.414969:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758819.414979:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655201408:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758819.542004:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758819.990084:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758820.565998:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758821.014137:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758821.590003:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758822.038078:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758822.613962:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758823.062113:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758823.638038:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758824.086079:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758824.534037:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758824.534050:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758824.534053:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758824.534062:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11880 00000100:00080000:3.0:1787758824.534073:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758824.534080:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758824.534118:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655201536/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758824.534145:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758824.534149:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758824.534151:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758824.534156:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11880 00000100:00080000:3.0:1787758824.534159:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758824.534175:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758824.534199:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758824.534211:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655201536:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787758824.534220:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655201664/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758824.534231:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758824.534235:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758824.534237:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758824.534241:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11880 00000100:00080000:3.0:1787758824.534244:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758824.534249:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758824.534254:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758824.534263:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655201792/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758824.534272:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758824.534275:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758824.534277:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758824.534280:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11880 00000800:00000100:2.0:1787758824.534283:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758824.534284:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758824.534288:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758824.534292:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758824.534301:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655201920/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758824.534310:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758824.534313:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758824.534315:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758824.534319:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11880 00000100:00080000:3.0:1787758824.534321:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758824.534326:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758824.534331:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758824.534337:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758824/real 1787758824] req@ffff8c05a9f79180 x1874588655201536/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758824.534341:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655202048/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758824.534357:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655201536/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758824.534361:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758824.534365:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758824.534367:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758824.534368:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758824.534371:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11880 00000100:00080000:2.0:1787758824.534373:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787758824.534374:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787758824.534377:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655201536:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758824.534378:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758824.534382:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758824.534393:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655202176/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758824.534401:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655201664:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758824.534409:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758824.534412:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758824.534414:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758824.534418:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11880 00000100:00080000:3.0:1787758824.534421:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787758824.534423:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758824.534425:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758824.534430:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758824.534436:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758824/real 1787758824] req@ffff8c05a9f7bb80 x1874588655201664/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758824.534441:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655202304/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758824.534449:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655201664/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758824.534459:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758824.534463:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787758824.534463:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787758824.534467:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655201664:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758824.534467:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758824.534469:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787758824.534473:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655201792:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758824.534473:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11880 00000100:00080000:3.0:1787758824.534477:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758824.534481:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758824.534485:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787758824.534491:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758824.534495:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655202432/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758824.534502:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758824/real 1787758824] req@ffff8c05a9f79f80 x1874588655201792/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758824.534513:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655201792/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758824.534521:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758824.534524:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758824.534528:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655201792:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758824.534534:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655201920:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758824.534555:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758824.534564:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758824/real 1787758824] req@ffff8c05a9f7b100 x1874588655201920/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758824.534572:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655201920/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758824.534577:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758824.534579:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758824.534582:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655201920:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758824.534586:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655202048:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758824.534603:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758824.534614:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758824/real 1787758824] req@ffff8c05a9f7ad80 x1874588655202048/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758824.534627:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655202048/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758824.534636:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758824.534639:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758824.534643:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655202048:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758824.534660:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655202176:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758824.534675:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758824.534683:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758824/real 1787758824] req@ffff8c05a9f78e00 x1874588655202176/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758824.534691:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655202176/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758824.534696:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758824.534698:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758824.534700:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655202176:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758824.534703:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655202304:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758824.534713:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758824.534719:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758824/real 1787758824] req@ffff8c05a9f7b800 x1874588655202304/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758824.534727:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655202304/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758824.534732:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758824.534734:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758824.534735:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655202304:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758824.534738:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655202432:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758824.534747:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758824.534753:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758824/real 1787758824] req@ffff8c05a9f79500 x1874588655202432/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758824.534760:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655202432/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758824.534766:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758824.534767:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758824.534769:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655202432:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758824.661994:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758825.110119:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758825.685958:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758826.134104:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758826.709962:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758827.158110:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758827.734014:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758828.182047:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758828.757980:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758829.206126:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758829.654011:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758829.654023:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758829.654028:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758829.654038:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11886 00000100:00080000:0.0:1787758829.654050:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758829.654057:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758829.654124:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655202560/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758829.654162:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758829.654166:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758829.654168:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758829.654173:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11886 00000100:00080000:0.0:1787758829.654178:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758829.654183:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758829.654194:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758829.654210:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655202688/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758829.654214:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655202560:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758829.654221:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758829.654224:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758829.654226:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758829.654230:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11886 00000100:00080000:0.0:1787758829.654234:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758829.654237:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758829.654242:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758829.654251:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655202816/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758829.654260:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758829.654263:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758829.654265:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758829.654269:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11886 00000100:00080000:0.0:1787758829.654274:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758829.654279:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758829.654284:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787758829.654287:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758829.654293:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588655202944/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758829.654300:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758829.654304:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758829.654305:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758829.654309:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11886 00000100:00080000:0.0:1787758829.654312:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758829.654315:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758829.654320:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758829.654329:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588655203072/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758829.654336:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758829/real 1787758829] req@ffff8c05a9f77800 x1874588655202560/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758829.654338:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758829.654341:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758829.654344:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758829.654347:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11886 00000100:00080000:0.0:1787758829.654350:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758829.654354:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758829.654356:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655202560/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787758829.654359:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787758829.654368:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787758829.654369:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588655203200/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758829.654372:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758829.654376:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655202560:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758829.654384:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758829.654387:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758829.654389:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758829.654392:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11886 00000100:00080000:0.0:1787758829.654395:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758829.654399:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758829.654402:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655202688:192.168.201.107@tcp:38:kworker.0 00000008:00100000:0.0:1787758829.654403:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758829.654412:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588655203328/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758829.654422:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787758829.654424:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758829.654426:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758829.654428:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758829.654432:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11886 00000100:00080000:0.0:1787758829.654436:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787758829.654437:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758829/real 1787758829] req@ffff8c05a9f74a80 x1874588655202688/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758829.654439:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758829.654444:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758829.654450:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655202688/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758829.654454:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588655203456/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758829.654459:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758829.654462:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758829.654466:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655202688:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758829.654471:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655202816:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758829.654490:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758829.654502:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758829/real 1787758829] req@ffff8c05a9f75880 x1874588655202816/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758829.654515:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655202816/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758829.654526:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758829.654529:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758829.654539:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655202816:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758829.654543:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655202944:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758829.654561:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758829.654568:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758829/real 1787758829] req@ffff8c05a9f75500 x1874588655202944/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758829.654576:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588655202944/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758829.654582:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758829.654584:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758829.654586:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655202944:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758829.654589:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655203072:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758829.654600:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758829.654605:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758829/real 1787758829] req@ffff8c05a9f74000 x1874588655203072/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758829.654612:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588655203072/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758829.654618:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758829.654619:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758829.654622:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655203072:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758829.654626:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655203200:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758829.654636:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758829.654641:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758829/real 1787758829] req@ffff8c05a9f75180 x1874588655203200/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758829.654648:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588655203200/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758829.654653:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758829.654655:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758829.654656:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655203200:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758829.654669:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655203328:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758829.654679:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758829.654685:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758829/real 1787758829] req@ffff8c05a9f76d80 x1874588655203328/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758829.654692:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588655203328/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758829.654697:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758829.654700:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758829.654702:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655203328:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758829.654713:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655203456:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758829.654724:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758829.654731:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758829/real 1787758829] req@ffff8c05a9f77100 x1874588655203456/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758829.654739:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588655203456/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758829.654744:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758829.654746:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758829.654748:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655203456:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758829.781998:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758830.230106:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758830.805994:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758831.254107:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758831.829988:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758832.278108:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758832.853971:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758833.302103:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758833.878009:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758834.325995:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758834.774036:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758834.774051:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758834.774057:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758834.774071:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11891 00000100:00080000:0.0:1787758834.774082:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758834.774097:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758834.774138:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588655203584/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758834.774164:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758834.774167:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758834.774168:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758834.774172:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11891 00000100:00080000:0.0:1787758834.774174:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758834.774177:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758834.774186:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758834.774197:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588655203712/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758834.774204:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758834.774207:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758834.774210:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758834.774213:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11891 00000100:00080000:0.0:1787758834.774216:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758834.774219:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758834.774224:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758834.774232:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588655203840/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758834.774246:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758834.774250:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758834.774253:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787758834.774259:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655203584:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758834.774260:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11891 00000100:00080000:0.0:1787758834.774268:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758834.774277:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758834.774282:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758834.774295:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588655203968/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758834.774301:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758834.774304:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758834.774306:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758834.774309:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11891 00000100:00080000:0.0:1787758834.774311:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758834.774314:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758834.774318:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758834.774326:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588655204096/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758834.774333:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758834.774336:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758834.774337:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758834.774340:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11891 00000100:00080000:0.0:1787758834.774345:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787758834.774349:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758834.774353:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758834.774359:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758834.774372:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588655204224/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758834.774379:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758834.774382:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758834.774384:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758834.774388:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11891 00000100:00080000:0.0:1787758834.774391:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758834.774395:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758834.774399:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758834.774404:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758834/real 1787758834] req@ffff8c05a9f74e00 x1874588655203584/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758834.774410:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655204352/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758834.774453:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758834.774457:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787758834.774458:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588655203584/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758834.774460:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758834.774474:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11891 00000100:00080000:0.0:1787758834.774485:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758834.774494:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:2.0:1787758834.774498:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787758834.774499:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787758834.774509:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758834.774515:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655204480/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758834.774516:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655203584:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787758834.774539:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655203712:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758834.774587:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758834.774605:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758834/real 1787758834] req@ffff8c05a9f77100 x1874588655203712/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758834.774618:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588655203712/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758834.774624:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758834.774627:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758834.774631:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655203712:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758834.774659:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655203840:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758834.774682:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758834.774692:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758834/real 1787758834] req@ffff8c05a9f76d80 x1874588655203840/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758834.774700:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588655203840/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758834.774705:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758834.774708:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758834.774710:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655203840:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758834.774714:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655203968:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758834.774727:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758834.774733:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758834/real 1787758834] req@ffff8c05a9f75180 x1874588655203968/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758834.774740:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588655203968/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758834.774746:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758834.774748:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758834.774751:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655203968:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758834.774754:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655204096:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758834.774764:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758834.774770:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758834/real 1787758834] req@ffff8c05a9f74000 x1874588655204096/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758834.774777:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588655204096/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758834.774782:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758834.774784:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758834.774786:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655204096:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758834.774789:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655204224:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758834.774799:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758834.774805:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758834/real 1787758834] req@ffff8c05a9f75500 x1874588655204224/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758834.774811:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588655204224/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758834.774816:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758834.774819:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758834.774821:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655204224:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758834.774825:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655204352:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758834.774835:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758834.774841:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758834/real 1787758834] req@ffff8c05a9f75880 x1874588655204352/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758834.774847:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655204352/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758834.774853:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758834.774854:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758834.774856:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655204352:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758834.774859:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655204480:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758834.774869:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758834.774905:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758834/real 1787758834] req@ffff8c05a9f74a80 x1874588655204480/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758834.774914:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655204480/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758834.774919:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758834.774921:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758834.774923:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655204480:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758834.901962:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758835.350185:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758835.926013:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758836.374109:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787758836.950000:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758837.398193:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787758837.974019:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758838.422103:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787758838.998029:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758839.446221:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758839.894059:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758839.894074:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758839.894079:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758839.894088:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11896 00000100:00080000:3.0:1787758839.894101:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758839.894115:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758839.894155:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655204608/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758839.894181:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758839.894184:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758839.894185:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758839.894189:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11896 00000100:00080000:3.0:1787758839.894191:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758839.894194:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758839.894201:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758839.894210:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655204736/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758839.894220:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758839.894223:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758839.894225:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787758839.894231:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655204608:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758839.894232:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11896 00000100:00080000:3.0:1787758839.894237:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758839.894241:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758839.894244:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758839.894252:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655204864/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758839.894260:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758839.894263:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758839.894264:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758839.894267:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11896 00000100:00080000:3.0:1787758839.894270:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758839.894272:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758839.894276:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758839.894284:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655204992/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758839.894291:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758839.894294:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758839.894295:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758839.894297:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11896 00000100:00080000:3.0:1787758839.894300:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758839.894304:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:2.0:1787758839.894311:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:3.0:1787758839.894311:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758839.894323:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655205120/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758839.894331:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758839.894334:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758839.894335:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758839.894338:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11896 00000100:00080000:3.0:1787758839.894340:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758839.894342:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758839.894345:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758839.894353:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655205248/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758839.894358:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758839.894361:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758839.894362:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758839.894368:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11896 00000100:00100000:2.0:1787758839.894371:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758839/real 1787758839] req@ffff8c05a9f7a300 x1874588655204608/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758839.894371:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758839.894377:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758839.894382:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758839.894392:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655205376/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758839.894404:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655204608/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758839.894405:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758839.894412:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758839.894415:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758839.894421:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11896 00000100:00080000:3.0:1787758839.894424:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758839.894425:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758839.894428:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:2.0:1787758839.894432:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000008:00100000:3.0:1787758839.894433:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758839.894438:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655204608:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787758839.894443:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655205504/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758839.894469:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655204736:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758839.894501:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758839.894516:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758839/real 1787758839] req@ffff8c05a9f79880 x1874588655204736/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758839.894531:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655204736/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758839.894549:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758839.894553:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758839.894557:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655204736:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758839.894564:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655204864:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758839.894586:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758839.894598:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758839/real 1787758839] req@ffff8c05a9f78700 x1874588655204864/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758839.894617:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655204864/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758839.894625:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758839.894628:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758839.894631:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655204864:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758839.894637:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655204992:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758839.894659:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758839.894670:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758839/real 1787758839] req@ffff8c05a9f7b480 x1874588655204992/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758839.894692:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655204992/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758839.894700:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758839.894703:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758839.894706:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655204992:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758839.894712:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655205120:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758839.894733:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758839.894745:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758839/real 1787758839] req@ffff8c05a9f78380 x1874588655205120/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758839.894758:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655205120/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758839.894771:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758839.894774:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758839.894778:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655205120:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758839.894783:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655205248:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758839.894803:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758839.894814:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758839/real 1787758839] req@ffff8c05a9f78a80 x1874588655205248/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758839.894827:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655205248/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758839.894835:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758839.894839:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758839.894843:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655205248:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758839.894857:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655205376:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758839.894939:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758839.894954:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758839/real 1787758839] req@ffff8c05a9f7a680 x1874588655205376/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758839.894968:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655205376/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758839.894977:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758839.894980:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758839.894985:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655205376:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758839.894992:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655205504:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758839.895015:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758839.895027:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758839/real 1787758839] req@ffff8c05a9f7aa00 x1874588655205504/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758839.895038:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655205504/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758839.895045:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758839.895048:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758839.895051:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655205504:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758840.022008:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758840.470062:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787758841.046004:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758841.494070:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787758842.069987:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758842.518055:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787758843.094003:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758843.542145:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787758844.117975:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758844.566007:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758845.014023:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758845.014035:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758845.014041:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758845.014054:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11901 00000100:00080000:3.0:1787758845.014067:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758845.014076:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758845.014126:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655205632/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758845.014162:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758845.014166:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758845.014168:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758845.014173:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11901 00000100:00080000:3.0:1787758845.014178:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758845.014184:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758845.014194:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758845.014211:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655205632:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787758845.014214:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655205760/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758845.014230:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758845.014234:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758845.014236:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758845.014240:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11901 00000100:00080000:3.0:1787758845.014245:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758845.014250:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758845.014255:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758845.014269:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655205888/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758845.014281:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787758845.014286:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758845.014289:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758845.014293:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758845.014298:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11901 00000100:00080000:3.0:1787758845.014304:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758845.014309:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758845.014315:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758845.014331:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655206016/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758845.014334:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758845/real 1787758845] req@ffff8c05a9f7aa00 x1874588655205632/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758845.014366:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787758845.014367:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655205632/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758845.014377:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758845.014382:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758845.014386:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758845.014390:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11901 00000100:00080000:3.0:1787758845.014396:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758845.014401:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:2.0:1787758845.014402:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758845.014409:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655205632:192.168.201.107@tcp:250:kworker.0 00000008:00100000:3.0:1787758845.014410:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758845.014426:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655206144/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758845.014434:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655205760:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758845.014441:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758845.014447:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758845.014449:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758845.014455:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11901 00000100:00080000:3.0:1787758845.014462:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787758845.014463:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758845.014468:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758845.014476:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758845.014480:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758845/real 1787758845] req@ffff8c05a9f7a680 x1874588655205760/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758845.014495:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655205760/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758845.014495:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655206272/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758845.014528:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758845.014530:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758845.014535:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787758845.014537:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787758845.014540:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787758845.014543:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655205760:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758845.014548:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11901 00000100:00100000:2.0:1787758845.014551:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655205888:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758845.014556:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758845.014564:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:2.0:1787758845.014569:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:3.0:1787758845.014570:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758845.014579:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758845/real 1787758845] req@ffff8c05a9f78a80 x1874588655205888/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758845.014587:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655206400/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758845.014604:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655205888/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758845.014608:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758845.014618:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758845.014622:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758845.014625:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655205888:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758845.014629:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655206016:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758845.014642:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758845.014649:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758845/real 1787758845] req@ffff8c05a9f78380 x1874588655206016/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758845.014653:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758845.014659:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787758845.014664:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655206016/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758845.014667:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11901 00000100:00080000:3.0:1787758845.014676:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758845.014678:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758845.014686:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787758845.014686:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758845.014692:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655206016:192.168.201.107@tcp:8:kworker.0 00000008:00100000:3.0:1787758845.014697:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758845.014706:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655206144:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758845.014716:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655206528/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787758845.014725:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758845.014737:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758845/real 1787758845] req@ffff8c05a9f7b480 x1874588655206144/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758845.014747:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655206144/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758845.014752:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758845.014755:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758845.014756:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655206144:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758845.014761:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655206272:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758845.014772:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758845.014778:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758845/real 1787758845] req@ffff8c05a9f78700 x1874588655206272/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758845.014786:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655206272/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758845.014791:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758845.014793:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758845.014797:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655206272:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758845.014801:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655206400:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758845.014812:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758845.014818:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758845/real 1787758845] req@ffff8c05a9f79880 x1874588655206400/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758845.014838:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655206400/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758845.014843:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758845.014845:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758845.014847:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655206400:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758845.014856:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655206528:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758845.014868:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758845.014897:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758845/real 1787758845] req@ffff8c05a9f7a300 x1874588655206528/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758845.014906:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655206528/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758845.014912:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758845.014914:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758845.014916:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655206528:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787758845.141964:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758845.590136:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787758846.165961:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758846.614038:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787758847.189943:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758847.638168:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787758848.213952:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758848.662051:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787758849.237998:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758849.686120:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758850.133943:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758850.133955:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758850.133960:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758850.133971:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11906 00000100:00080000:3.0:1787758850.133982:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758850.133989:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758850.134030:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655206656/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758850.134056:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758850.134060:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758850.134061:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758850.134065:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11906 00000100:00080000:3.0:1787758850.134067:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758850.134070:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758850.134078:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758850.134087:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655206784/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758850.134095:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758850.134097:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758850.134098:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758850.134101:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11906 00000100:00080000:3.0:1787758850.134104:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758850.134106:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758850.134109:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758850.134117:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655206912/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758850.134122:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655206656:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758850.134128:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758850.134131:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758850.134132:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758850.134135:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11906 00000100:00080000:3.0:1787758850.134137:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758850.134140:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758850.134144:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758850.134153:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655207040/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758850.134160:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758850.134162:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758850.134164:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758850.134166:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11906 00000100:00080000:3.0:1787758850.134184:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758850.134189:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758850.134195:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787758850.134198:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758850.134215:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655207168/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758850.134226:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758850.134229:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758850.134230:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758850.134234:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11906 00000100:00080000:3.0:1787758850.134238:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758850.134242:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758850.134247:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758850/real 1787758850] req@ffff8c05a9f79500 x1874588655206656/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787758850.134249:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758850.134264:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655207296/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758850.134270:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655206656/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758850.134276:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758850.134283:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758850.134287:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758850.134291:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758850.134292:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11906 00000100:00080000:3.0:1787758850.134298:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758850.134301:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787758850.134306:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758850.134309:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655206656:192.168.201.107@tcp:250:kworker.0 00000008:00100000:3.0:1787758850.134316:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758850.134318:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655206784:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758850.134335:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655207424/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787758850.134342:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758850.134355:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758850.134358:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758850.134360:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758850.134364:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11906 00000100:00100000:2.0:1787758850.134366:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758850/real 1787758850] req@ffff8c05a9f7b800 x1874588655206784/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758850.134367:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758850.134376:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758850.134385:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758850.134386:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655206784/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758850.134397:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758850.134400:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758850.134403:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655206784:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758850.134407:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655207552/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758850.134430:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655206912:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758850.134447:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758850.134455:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758850/real 1787758850] req@ffff8c05a9f78e00 x1874588655206912/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758850.134464:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655206912/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758850.134474:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758850.134476:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758850.134479:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655206912:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758850.134482:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655207040:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758850.134502:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758850.134511:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758850/real 1787758850] req@ffff8c05a9f7ad80 x1874588655207040/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758850.134519:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655207040/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758850.134525:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758850.134527:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758850.134530:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655207040:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758850.134535:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655207168:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758850.134555:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758850.134563:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758850/real 1787758850] req@ffff8c05a9f7b100 x1874588655207168/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758850.134571:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655207168/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758850.134577:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758850.134580:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758850.134583:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655207168:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758850.134588:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655207296:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758850.134602:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758850.134608:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758850/real 1787758850] req@ffff8c05a9f79f80 x1874588655207296/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758850.134625:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655207296/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758850.134631:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758850.134634:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758850.134637:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655207296:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758850.134641:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655207424:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758850.134655:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758850.134662:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758850/real 1787758850] req@ffff8c05a9f7bb80 x1874588655207424/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758850.134671:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655207424/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758850.134676:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758850.134678:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758850.134680:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655207424:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758850.134684:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655207552:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758850.134704:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758850.134712:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758850/real 1787758850] req@ffff8c05a9f79180 x1874588655207552/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758850.134720:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655207552/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758850.134727:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758850.134747:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758850.134750:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655207552:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758850.262011:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758850.710028:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758851.286023:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758851.734134:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758852.309956:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758852.758001:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758853.333997:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758853.782166:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758854.357951:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758854.806073:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758855.254045:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758855.254059:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758855.254065:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758855.254080:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11911 00000100:00080000:0.0:1787758855.254094:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758855.254104:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758855.254149:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655207680/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758855.254186:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758855.254190:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758855.254192:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758855.254198:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11911 00000100:00080000:0.0:1787758855.254202:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758855.254208:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758855.254218:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758855.254220:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655207680:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787758855.254237:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655207808/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758855.254250:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758855.254255:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758855.254257:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758855.254262:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11911 00000100:00080000:0.0:1787758855.254267:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758855.254271:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758855.254277:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758855.254290:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655207936/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787758855.254294:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758855.254308:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758855.254313:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758855.254316:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758855.254322:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11911 00000100:00080000:0.0:1787758855.254328:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758855.254333:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758855.254339:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758855/real 1787758855] req@ffff8c05a9f77800 x1874588655207680/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787758855.254340:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758855.254358:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655208064/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758855.254362:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655207680/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758855.254387:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758855.254388:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758855.254395:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787758855.254397:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1787758855.254402:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655207680:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758855.254403:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758855.254412:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11911 00000100:00080000:0.0:1787758855.254417:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758855.254422:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758855.254427:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758855.254429:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655207808:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758855.254445:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588655208192/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787758855.254454:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758855.254465:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787758855.254467:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758855/real 1787758855] req@ffff8c05a9f74380 x1874588655207808/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758855.254474:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758855.254478:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787758855.254484:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655207808/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758855.254485:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11911 00000100:00080000:0.0:1787758855.254492:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758855.254495:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758855.254500:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:2.0:1787758855.254502:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758855.254509:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655207808:192.168.201.107@tcp:38:kworker.0 00000008:00100000:0.0:1787758855.254511:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758855.254515:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655207936:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758855.254532:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588655208320/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787758855.254535:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758855.254545:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758855/real 1787758855] req@ffff8c05a9f76680 x1874588655207936/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758855.254575:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787758855.254575:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655207936/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758855.254585:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758855.254591:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758855.254591:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758855.254599:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787758855.254603:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11911 00000100:00100000:2.0:1787758855.254604:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655207936:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758855.254613:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787758855.254614:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655208064:192.168.201.107@tcp:8:kworker.0 00000100:00080000:0.0:1787758855.254623:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758855.254630:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787758855.254634:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758855.254664:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588655208448/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758855.254667:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758855/real 1787758855] req@ffff8c05a9f76300 x1874588655208064/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758855.254693:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787758855.254695:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655208064/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758855.254700:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758855.254704:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758855.254711:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758855.254713:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11911 00000100:00080000:2.0:1787758855.254717:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787758855.254721:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787758855.254721:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655208064:192.168.201.107@tcp:8:kworker.0 00000100:00080000:0.0:1787758855.254732:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758855.254739:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758855.254740:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655208192:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758855.254761:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655208576/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787758855.254765:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758855.254776:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758855/real 1787758855] req@ffff8c05a9f77b80 x1874588655208192/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758855.254784:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588655208192/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758855.254794:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758855.254797:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758855.254800:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655208192:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758855.254806:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655208320:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758855.254826:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758855.254841:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758855/real 1787758855] req@ffff8c05a9f76a00 x1874588655208320/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758855.254856:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588655208320/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758855.254862:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758855.254864:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758855.254867:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655208320:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758855.254911:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655208448:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758855.254934:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758855.254942:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758855/real 1787758855] req@ffff8c05a9f75c00 x1874588655208448/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758855.254950:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588655208448/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758855.254956:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758855.254958:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758855.254960:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655208448:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758855.254970:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655208576:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758855.254981:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758855.254986:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758855/real 1787758855] req@ffff8c05a9f76300 x1874588655208576/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758855.254994:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655208576/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758855.254999:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758855.255001:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758855.255003:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655208576:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758855.381985:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758855.830128:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758856.406035:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758856.854079:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758857.429972:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758857.878132:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758858.453952:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758858.902095:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758859.477989:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758859.926142:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758860.374025:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758860.374036:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758860.374041:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758860.374051:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11916 00000100:00080000:1.0:1787758860.374062:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758860.374069:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758860.374112:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655208704/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758860.374141:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758860.374144:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758860.374145:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758860.374148:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11916 00000100:00080000:1.0:1787758860.374151:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758860.374154:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758860.374163:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758860.374173:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655208832/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758860.374188:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758860.374192:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787758860.374194:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655208704:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758860.374195:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758860.374204:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11916 00000100:00080000:1.0:1787758860.374207:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758860.374209:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758860.374213:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758860.374225:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655208960/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758860.374232:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758860.374235:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758860.374236:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758860.374238:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11916 00000100:00080000:1.0:1787758860.374240:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758860.374243:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758860.374247:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758860.374255:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655209088/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758860.374266:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787758860.374271:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758860.374272:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758860.374275:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758860.374279:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11916 00000100:00080000:1.0:1787758860.374283:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758860.374286:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758860.374289:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758860.374299:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655209216/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758860.374305:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758860.374307:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758860.374309:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758860.374312:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11916 00000100:00080000:1.0:1787758860.374314:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758860.374319:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758860.374319:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758860/real 1787758860] req@ffff8c05b2d99880 x1874588655208704/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787758860.374322:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758860.374334:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655209344/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758860.374341:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787758860.374342:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655208704/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758860.374344:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758860.374348:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758860.374356:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11916 00000100:00080000:2.0:1787758860.374361:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758860.374362:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758860.374368:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:2.0:1787758860.374370:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000008:00100000:1.0:1787758860.374373:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758860.374377:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655208704:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758860.374385:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655209472/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758860.374392:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758860.374394:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758860.374395:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758860.374397:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11916 00000100:00080000:1.0:1787758860.374399:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758860.374406:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758860.374406:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655208832:192.168.201.107@tcp:38:kworker.0 00000008:00100000:1.0:1787758860.374413:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758860.374424:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655209600/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787758860.374435:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758860.374446:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758860/real 1787758860] req@ffff8c05b2d9bb80 x1874588655208832/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758860.374456:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655208832/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758860.374463:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758860.374466:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758860.374468:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655208832:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758860.374482:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655208960:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758860.374498:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758860.374506:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758860/real 1787758860] req@ffff8c05b2d9b100 x1874588655208960/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758860.374514:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655208960/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758860.374520:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758860.374522:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758860.374524:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655208960:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758860.374528:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655209088:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758860.374542:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758860.374567:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758860/real 1787758860] req@ffff8c05b2d98000 x1874588655209088/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758860.374581:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655209088/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758860.374591:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758860.374594:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758860.374598:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655209088:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758860.374605:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655209216:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758860.374633:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758860.374642:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758860/real 1787758860] req@ffff8c05b2d98700 x1874588655209216/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758860.374656:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655209216/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758860.374662:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758860.374664:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758860.374666:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655209216:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758860.374670:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655209344:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758860.374682:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758860.374688:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758860/real 1787758860] req@ffff8c05b2d9b480 x1874588655209344/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758860.374696:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655209344/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758860.374701:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758860.374703:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758860.374705:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655209344:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758860.374708:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655209472:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758860.374718:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758860.374723:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758860/real 1787758860] req@ffff8c05b2d9aa00 x1874588655209472/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758860.374731:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655209472/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758860.374741:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758860.374743:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758860.374745:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655209472:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758860.374759:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655209600:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758860.374769:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758860.374774:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758860/real 1787758860] req@ffff8c05b2d99f80 x1874588655209600/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758860.374781:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655209600/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758860.374787:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758860.374789:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758860.374791:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655209600:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758860.502044:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758860.950104:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758861.525959:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758861.974181:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758862.549969:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758862.998069:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758863.573960:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758864.022102:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758864.598011:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758865.046081:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758865.493984:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758865.493994:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787758865.493999:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758865.494009:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11921 00000100:00080000:2.0:1787758865.494019:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758865.494026:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758865.494060:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588655209728/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758865.494087:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758865.494090:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758865.494091:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758865.494094:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11921 00000100:00080000:2.0:1787758865.494096:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758865.494100:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758865.494109:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758865.494128:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588655209856/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787758865.494138:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655209728:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787758865.494157:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758865.494163:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758865.494166:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758865.494172:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11921 00000100:00080000:2.0:1787758865.494175:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758865.494180:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758865.494186:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758865.494196:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588655209984/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758865.494202:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758865.494204:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758865.494205:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758865.494207:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11921 00000100:00080000:2.0:1787758865.494209:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758865.494211:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758865.494215:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758865.494226:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588655210112/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758865.494233:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758865.494235:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758865.494236:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758865.494238:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11921 00000100:00080000:2.0:1787758865.494239:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758865.494242:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758865.494247:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758865.494253:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588655210240/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758865.494269:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758865.494273:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758865.494274:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758865.494276:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11921 00000100:00080000:2.0:1787758865.494278:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758865.494281:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758865.494284:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758865.494290:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588655210368/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758865.494294:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758865.494296:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787758865.494298:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758865.494300:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11921 00000100:00080000:2.0:1787758865.494302:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758865.494304:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758865.494308:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758865.494316:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588655210496/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758865.494338:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758865.494345:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758865.494357:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787758865.494358:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655209856:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787758865.494367:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11921 00000100:00080000:2.0:1787758865.494376:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758865.494379:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758865.494382:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758865.494394:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588655210624/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787758865.494450:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655209984:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758865.494493:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655210112:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758865.494522:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655210240:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758865.494563:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655210368:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758865.494605:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655210496:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758865.494651:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655210624:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758865.621951:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758866.070118:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758866.645982:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758867.094075:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758867.669970:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758868.118079:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787758868.566185:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.107:988 00000400:00000100:0.0:1787758868.566238:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.107@tcp at host 192.168.201.107 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787758868.566258:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787758868.567813:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787758868.567824:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787758868.567829:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787758868.567834:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787758868.567838:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787758868.567843:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787758868.567847:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:3.0:1787758868.567961:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758865/real 1787758868] req@ffff8c05b2e70380 x1874588655210624/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758868.568009:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588655210624/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758868.568022:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758868.568028:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758868.568033:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655210624:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758868.568042:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758865/real 1787758868] req@ffff8c05b2e70700 x1874588655209856/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758868.568061:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588655209856/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758868.568068:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758868.568072:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758868.568076:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655209856:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758868.568083:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758865/real 1787758868] req@ffff8c05b2e73480 x1874588655209984/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758868.568097:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588655209984/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758868.568103:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758868.568107:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758868.568110:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655209984:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758868.568118:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758865/real 1787758868] req@ffff8c05b2e71500 x1874588655210112/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758868.568129:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588655210112/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758868.568138:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758868.568141:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758868.568145:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655210112:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758868.568151:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758865/real 1787758868] req@ffff8c05b2e73b80 x1874588655210240/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758868.568160:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588655210240/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758868.568166:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758868.568168:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758868.568176:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655210240:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758868.568183:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758865/real 1787758868] req@ffff8c05b2e71c00 x1874588655210368/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758868.568194:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588655210368/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758868.568201:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758868.568204:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758868.568208:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655210368:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758868.568215:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758865/real 1787758868] req@ffff8c05b2e71880 x1874588655210496/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758868.568227:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588655210496/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758868.568233:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758868.568236:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758868.568238:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655210496:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758868.568242:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758865/real 1787758868] req@ffff8c05b2e70e00 x1874588655209728/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758868.568250:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588655209728/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758868.568255:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758868.568257:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758868.568258:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655209728:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787758868.694015:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758869.141999:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758869.718008:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758870.166108:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758870.613990:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758870.614004:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787758870.614009:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758870.614020:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11927 00000100:00080000:2.0:1787758870.614033:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758870.614042:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758870.614089:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73800 x1874588655210752/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758870.614130:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758870.614134:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758870.614137:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758870.614142:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11927 00000100:00080000:2.0:1787758870.614146:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758870.614151:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758870.614171:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758870.614190:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588655210880/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758870.614204:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787758870.614204:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655210752:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787758870.614213:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758870.614217:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758870.614222:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11927 00000100:00080000:2.0:1787758870.614226:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758870.614231:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758870.614236:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758870.614247:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588655211008/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758870.614260:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758870.614264:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758870.614266:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758870.614271:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11927 00000100:00080000:2.0:1787758870.614276:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758870.614281:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758870.614287:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787758870.614295:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758870.614301:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588655211136/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758870.614314:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758870.614318:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758870.614320:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758870.614331:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11927 00000100:00080000:2.0:1787758870.614337:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758870.614342:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758870.614348:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758870.614349:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758870/real 1787758870] req@ffff8c05b2e73800 x1874588655210752/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758870.614366:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588655211264/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758870.614396:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787758870.614398:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73800 x1874588655210752/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758870.614404:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758870.614409:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758870.614418:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11927 00000100:00080000:3.0:1787758870.614421:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758870.614423:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758870.614431:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:3.0:1787758870.614431:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000008:00100000:2.0:1787758870.614442:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758870.614442:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655210752:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787758870.614460:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588655211392/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758870.614483:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758870.614487:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787758870.614489:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758870.614493:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11927 00000100:00080000:2.0:1787758870.614498:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758870.614502:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758870.614508:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758870.614511:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655210880:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758870.614525:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588655211520/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758870.614539:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758870.614543:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758870.614545:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787758870.614549:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787758870.614553:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11927 00000100:00080000:2.0:1787758870.614560:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758870.614566:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758870.614571:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758870/real 1787758870] req@ffff8c05b2e70e00 x1874588655210880/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787758870.614574:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758870.614591:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588655210880/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758870.614594:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588655211648/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758870.614627:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758870.614633:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758870.614638:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655210880:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758870.614648:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655211008:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758870.614671:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758870.614683:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758870/real 1787758870] req@ffff8c05b2e71880 x1874588655211008/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758870.614696:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588655211008/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758870.614703:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758870.614707:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758870.614710:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655211008:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758870.614716:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655211136:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758870.614822:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758870.614836:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758870/real 1787758870] req@ffff8c05b2e71c00 x1874588655211136/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758870.614850:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588655211136/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758870.614859:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758870.614862:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758870.614867:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655211136:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758870.614873:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655211264:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758870.614925:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758870.614938:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758870/real 1787758870] req@ffff8c05b2e73b80 x1874588655211264/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758870.614951:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588655211264/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758870.614960:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758870.614964:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758870.614967:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655211264:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758870.614973:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655211392:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758870.614994:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758870.615006:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758870/real 1787758870] req@ffff8c05b2e71500 x1874588655211392/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758870.615028:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588655211392/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758870.615037:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758870.615041:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758870.615045:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655211392:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758870.615066:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655211520:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758870.615088:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758870.615101:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758870/real 1787758870] req@ffff8c05b2e73480 x1874588655211520/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758870.615113:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588655211520/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758870.615122:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758870.615125:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758870.615128:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655211520:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758870.615135:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655211648:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758870.615157:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758870.615168:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758870/real 1787758870] req@ffff8c05b2e70700 x1874588655211648/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758870.615181:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588655211648/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758870.615196:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758870.615200:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758870.615204:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655211648:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758870.741977:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758871.190119:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758871.765969:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758872.214071:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758872.790017:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758873.238512:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758873.813982:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758874.262133:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758874.837981:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758875.286100:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758875.733975:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758875.733986:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787758875.733991:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758875.734002:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11932 00000100:00080000:2.0:1787758875.734015:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758875.734022:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758875.734065:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588655211776/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758875.734088:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758875.734091:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758875.734092:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758875.734095:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11932 00000100:00080000:2.0:1787758875.734097:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758875.734100:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758875.734108:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758875.734118:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588655211904/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758875.734127:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758875.734129:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758875.734130:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758875.734132:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11932 00000100:00080000:2.0:1787758875.734135:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787758875.734139:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655211776:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787758875.734140:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758875.734148:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758875.734157:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588655212032/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758875.734164:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758875.734167:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758875.734168:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758875.734171:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11932 00000100:00080000:2.0:1787758875.734174:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758875.734176:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758875.734179:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758875.734187:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588655212160/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758875.734192:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758875.734195:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758875.734196:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758875.734198:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11932 00000100:00080000:2.0:1787758875.734202:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758875.734205:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758875.734208:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758875.734215:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588655212288/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787758875.734220:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787758875.734229:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758875.734232:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758875.734234:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758875.734238:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11932 00000100:00080000:2.0:1787758875.734241:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758875.734244:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758875.734248:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758875.734257:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588655212416/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758875.734263:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758875.734265:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787758875.734266:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758875.734269:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11932 00000100:00080000:2.0:1787758875.734271:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758875.734274:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758875.734296:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758875/real 1787758875] req@ffff8c05b2e70380 x1874588655211776/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787758875.734299:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758875.734320:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588655212544/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787758875.734324:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588655211776/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758875.734346:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758875.734350:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758875.734351:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758875.734352:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758875.734362:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787758875.734363:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11932 00000100:00080000:2.0:1787758875.734371:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787758875.734373:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655211776:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787758875.734380:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758875.734388:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758875.734403:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588655212672/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787758875.734409:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655211904:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758875.734444:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758875.734460:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758875/real 1787758875] req@ffff8c05b2e71180 x1874588655211904/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758875.734473:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588655211904/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758875.734481:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758875.734485:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758875.734488:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655211904:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758875.734494:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655212032:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758875.734515:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758875.734527:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758875/real 1787758875] req@ffff8c05b2e72680 x1874588655212032/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758875.734548:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588655212032/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758875.734557:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758875.734561:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758875.734565:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655212032:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758875.734570:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655212160:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758875.734588:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758875.734596:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758875/real 1787758875] req@ffff8c05b2e72a00 x1874588655212160/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758875.734605:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588655212160/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758875.734610:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758875.734613:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758875.734615:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655212160:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758875.734618:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655212288:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758875.734628:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758875.734634:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758875/real 1787758875] req@ffff8c05b2e72300 x1874588655212288/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758875.734641:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588655212288/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758875.734651:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758875.734654:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758875.734656:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655212288:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758875.734659:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655212416:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758875.734669:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758875.734675:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758875/real 1787758875] req@ffff8c05b2e70a80 x1874588655212416/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758875.734687:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588655212416/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758875.734696:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758875.734699:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758875.734703:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655212416:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758875.734709:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655212544:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758875.734729:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758875.734741:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758875/real 1787758875] req@ffff8c05b2e70000 x1874588655212544/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758875.734749:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588655212544/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758875.734755:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758875.734757:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758875.734760:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655212544:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758875.734775:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655212672:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758875.734788:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758875.734795:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758875/real 1787758875] req@ffff8c05b2e73100 x1874588655212672/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758875.734803:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588655212672/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758875.734808:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758875.734810:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758875.734812:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655212672:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758875.862014:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758876.310094:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758876.885963:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758877.334037:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758877.909996:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758878.358167:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758878.933967:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758879.382086:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758879.957996:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758880.406164:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758880.854031:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758880.854040:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758880.854044:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758880.854052:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11937 00000100:00080000:3.0:1787758880.854062:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758880.854068:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758880.854105:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655212800/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758880.854139:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758880.854142:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758880.854143:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758880.854147:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11937 00000100:00080000:3.0:1787758880.854149:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758880.854152:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758880.854160:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758880.854170:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655212928/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758880.854177:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758880.854180:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758880.854181:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758880.854186:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11937 00000100:00100000:2.0:1787758880.854199:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655212800:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758880.854200:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758880.854209:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758880.854215:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758880.854226:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655213056/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758880.854233:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758880.854236:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758880.854237:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758880.854241:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11937 00000100:00080000:3.0:1787758880.854244:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758880.854246:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758880.854250:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758880.854258:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655213184/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758880.854272:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787758880.854277:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758880.854278:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758880.854282:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758880.854286:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11937 00000100:00080000:3.0:1787758880.854290:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758880.854293:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758880.854297:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758880.854305:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655213312/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758880.854311:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758880.854314:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758880.854315:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758880.854318:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11937 00000100:00080000:3.0:1787758880.854320:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758880.854322:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758880.854325:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758880.854334:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758880/real 1787758880] req@ffff8c05a9f79180 x1874588655212800/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758880.854335:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655213440/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758880.854369:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758880.854372:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787758880.854375:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655212800/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758880.854376:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758880.854387:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11937 00000100:00080000:3.0:1787758880.854391:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758880.854392:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758880.854397:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:2.0:1787758880.854400:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000008:00100000:3.0:1787758880.854406:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758880.854407:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655212800:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787758880.854416:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655213568/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758880.854423:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758880.854426:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758880.854427:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758880.854430:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11937 00000100:00080000:3.0:1787758880.854432:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758880.854435:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758880.854437:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758880.854444:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655212928:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758880.854446:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655213696/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787758880.854470:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758880.854483:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758880/real 1787758880] req@ffff8c05a9f7bb80 x1874588655212928/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758880.854493:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655212928/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758880.854499:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758880.854502:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758880.854504:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655212928:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758880.854508:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655213056:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758880.854519:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758880.854525:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758880/real 1787758880] req@ffff8c05a9f79f80 x1874588655213056/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758880.854538:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655213056/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758880.854543:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758880.854544:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758880.854546:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655213056:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758880.854550:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655213184:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758880.854560:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758880.854566:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758880/real 1787758880] req@ffff8c05a9f7b100 x1874588655213184/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758880.854574:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655213184/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758880.854579:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758880.854581:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758880.854583:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655213184:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758880.854586:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655213312:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758880.854596:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758880.854602:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758880/real 1787758880] req@ffff8c05a9f7ad80 x1874588655213312/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758880.854609:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655213312/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758880.854619:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758880.854621:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758880.854623:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655213312:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758880.854626:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655213440:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758880.854637:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758880.854642:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758880/real 1787758880] req@ffff8c05a9f78e00 x1874588655213440/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758880.854648:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655213440/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758880.854653:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758880.854655:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758880.854657:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655213440:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758880.854660:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655213568:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758880.854670:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758880.854675:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758880/real 1787758880] req@ffff8c05a9f7b800 x1874588655213568/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758880.854682:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655213568/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758880.854687:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758880.854689:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758880.854691:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655213568:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758880.854704:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655213696:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758880.854714:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758880.854720:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758880/real 1787758880] req@ffff8c05a9f79500 x1874588655213696/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758880.854727:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655213696/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758880.854732:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758880.854734:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758880.854736:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655213696:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758880.981958:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758881.430082:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758882.005983:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758882.454208:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758883.029954:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758883.478006:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758884.054013:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758884.502167:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758885.077967:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758885.526044:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758885.973971:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758885.973984:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758885.973989:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758885.973999:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11942 00000100:00080000:3.0:1787758885.974011:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758885.974019:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758885.974059:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655213824/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758885.974094:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758885.974110:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758885.974112:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758885.974118:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11942 00000100:00080000:3.0:1787758885.974123:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758885.974128:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758885.974139:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758885.974144:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655213824:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787758885.974159:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655213952/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758885.974173:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758885.974177:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758885.974180:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758885.974185:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11942 00000100:00080000:3.0:1787758885.974199:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758885.974204:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:2.0:1787758885.974209:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:3.0:1787758885.974215:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758885.974250:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758885/real 1787758885] req@ffff8c05a9f7a300 x1874588655213824/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758885.974265:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655213824/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758885.974274:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758885.974277:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758885.974280:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655213824:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787758885.974303:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655214080/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758885.974305:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655213952:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758885.974325:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787758885.974329:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758885.974340:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758885/real 1787758885] req@ffff8c05a9f79880 x1874588655213952/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758885.974348:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655213952/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758885.974354:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758885.974357:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758885.974359:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655213952:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758885.974366:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655214080:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758885.974378:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758885.974385:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758885/real 1787758885] req@ffff8c05a9f78700 x1874588655214080/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758885.974400:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655214080/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758885.974408:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758885.974412:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758885.974416:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655214080:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758885.974446:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758885.974449:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758885.974456:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11942 00000100:00080000:3.0:1787758885.974461:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758885.974467:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758885.974475:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758885.974491:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655214208/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758885.974565:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758885.974575:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758885.974577:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787758885.974580:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655214208:192.168.201.107@tcp:8:kworker.0 00000100:00080000:3.0:1787758885.974585:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11942 00000100:00080000:3.0:1787758885.974592:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758885.974599:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758885.974606:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787758885.974609:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758885.974623:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758885/real 1787758885] req@ffff8c05a9f7b480 x1874588655214208/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758885.974628:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655214336/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758885.974637:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655214208/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758885.974651:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758885.974652:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758885.974658:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787758885.974661:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787758885.974664:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655214208:192.168.201.107@tcp:8:kworker.0 00000100:00080000:3.0:1787758885.974666:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758885.974673:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11942 00000100:00100000:2.0:1787758885.974678:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655214336:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758885.974679:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758885.974688:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758885.974695:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787758885.974700:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758885.974713:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758885/real 1787758885] req@ffff8c05a9f78380 x1874588655214336/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758885.974715:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655214464/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758885.974737:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655214336/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758885.974742:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758885.974747:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758885.974748:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787758885.974752:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787758885.974753:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787758885.974757:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655214336:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758885.974765:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655214464:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758885.974771:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11942 00000100:00080000:3.0:1787758885.974778:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787758885.974782:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758885.974788:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758885.974793:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758885/real 1787758885] req@ffff8c05a9f78a80 x1874588655214464/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787758885.974796:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758885.974807:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655214464/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758885.974816:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655214592/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758885.974836:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758885.974842:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758885.974842:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758885.974845:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758885.974851:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787758885.974853:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11942 00000100:00080000:3.0:1787758885.974860:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787758885.974862:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655214464:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758885.974868:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758885.974898:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758885.974913:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655214592:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758885.974915:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655214720/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787758885.974941:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758885.974953:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758885/real 1787758885] req@ffff8c05a9f7a680 x1874588655214592/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758885.974962:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655214592/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758885.974968:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758885.974970:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758885.974972:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655214592:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758885.974979:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655214720:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758885.974992:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758885.975003:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758885/real 1787758885] req@ffff8c05a9f7aa00 x1874588655214720/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758885.975015:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655214720/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758885.975024:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758885.975027:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758885.975030:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655214720:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758886.102018:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758886.550111:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758887.125962:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758887.574032:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758888.151109:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758888.598209:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758889.173997:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758889.622118:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758890.197965:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758890.646131:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758891.093987:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758891.093999:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758891.094004:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758891.094041:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11947 00000100:00080000:3.0:1787758891.094053:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758891.094063:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758891.094114:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655214848/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758891.094149:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758891.094153:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758891.094155:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758891.094160:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11947 00000100:00080000:3.0:1787758891.094165:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758891.094170:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758891.094180:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758891.094198:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655214976/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758891.094208:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655214848:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758891.094214:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758891.094219:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758891.094222:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758891.094227:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11947 00000100:00080000:3.0:1787758891.094241:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758891.094246:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758891.094253:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758891.094267:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655215104/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758891.094278:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787758891.094282:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758891.094285:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758891.094289:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758891.094295:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11947 00000100:00080000:3.0:1787758891.094300:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758891.094314:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758891.094320:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758891.094336:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655215232/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758891.094341:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758891/real 1787758891] req@ffff8c05a9f7aa00 x1874588655214848/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758891.094357:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758891.094363:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758891.094365:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787758891.094371:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655214848/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758891.094372:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11947 00000100:00080000:3.0:1787758891.094380:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758891.094389:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758891.094389:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:2.0:1787758891.094398:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000008:00100000:3.0:1787758891.094399:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758891.094406:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655214848:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787758891.094418:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655215360/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758891.094430:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758891.094434:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758891.094436:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758891.094440:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11947 00000100:00080000:3.0:1787758891.094443:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758891.094448:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758891.094453:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758891.094465:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655215488/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758891.094473:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655214976:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758891.094483:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758891.094488:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758891.094490:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787758891.094503:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758891.094516:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758891/real 1787758891] req@ffff8c05a9f7a680 x1874588655214976/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758891.094520:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11947 00000100:00080000:3.0:1787758891.094529:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758891.094537:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758891.094540:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655214976/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787758891.094545:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787758891.094555:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758891.094560:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758891.094567:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655215616/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758891.094568:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655214976:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758891.094581:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655215104:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787758891.094588:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758891.094593:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758891.094595:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758891.094601:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11947 00000100:00080000:3.0:1787758891.094606:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787758891.094607:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758891.094615:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758891.094622:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758891/real 1787758891] req@ffff8c05a9f78a80 x1874588655215104/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787758891.094622:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758891.094640:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655215744/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758891.094655:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655215104/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758891.094665:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758891.094668:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758891.094671:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655215104:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758891.094680:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655215232:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758891.094699:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758891.094709:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758891/real 1787758891] req@ffff8c05a9f78380 x1874588655215232/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758891.094721:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655215232/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758891.094730:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758891.094733:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758891.094737:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655215232:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758891.094743:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655215360:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758891.094759:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758891.094768:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758891/real 1787758891] req@ffff8c05a9f7b480 x1874588655215360/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758891.094780:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655215360/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758891.094795:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758891.094798:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758891.094802:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655215360:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758891.094817:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655215488:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758891.094837:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758891.094849:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758891/real 1787758891] req@ffff8c05a9f78700 x1874588655215488/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758891.094861:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655215488/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758891.094870:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758891.094903:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758891.094908:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655215488:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758891.094915:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655215616:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758891.094934:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758891.094945:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758891/real 1787758891] req@ffff8c05a9f79880 x1874588655215616/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758891.094957:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655215616/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758891.094966:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758891.094969:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758891.094973:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655215616:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758891.094979:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655215744:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758891.094995:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758891.095004:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758891/real 1787758891] req@ffff8c05a9f7a300 x1874588655215744/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758891.095034:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655215744/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758891.095044:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758891.095047:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758891.095050:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655215744:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758891.221979:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758891.670113:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758892.246050:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758892.694148:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758893.270014:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758893.718079:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758894.294035:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758894.742113:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758895.318066:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758895.766116:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758896.213989:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758896.213999:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758896.214004:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758896.214014:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11952 00000100:00080000:3.0:1787758896.214024:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758896.214031:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758896.214070:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655215872/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758896.214098:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758896.214101:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758896.214103:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758896.214106:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11952 00000100:00080000:3.0:1787758896.214108:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758896.214111:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758896.214120:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758896.214130:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655216000/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758896.214138:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758896.214141:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758896.214142:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758896.214145:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11952 00000100:00080000:3.0:1787758896.214147:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758896.214149:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758896.214152:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758896.214157:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655215872:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787758896.214162:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655216128/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758896.214171:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758896.214174:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758896.214175:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758896.214177:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11952 00000100:00080000:3.0:1787758896.214179:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758896.214182:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758896.214185:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758896.214192:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655216256/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758896.214198:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758896.214201:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758896.214202:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758896.214204:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11952 00000100:00080000:3.0:1787758896.214206:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758896.214209:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758896.214213:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758896.214221:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655216384/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758896.214226:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758896.214229:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758896.214230:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758896.214233:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11952 00000100:00080000:3.0:1787758896.214248:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758896.214250:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758896.214253:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787758896.214264:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758896.214265:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655216512/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758896.214278:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758896.214281:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758896.214283:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758896.214286:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11952 00000100:00080000:3.0:1787758896.214289:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758896.214291:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758896.214294:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758896.214316:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655216640/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758896.214320:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758896/real 1787758896] req@ffff8c05a9f79500 x1874588655215872/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758896.214346:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758896.214351:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758896.214353:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787758896.214360:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655215872/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758896.214360:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11952 00000100:00080000:3.0:1787758896.214366:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758896.214376:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:2.0:1787758896.214382:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787758896.214384:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787758896.214388:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758896.214394:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655215872:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787758896.214402:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655216768/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758896.214404:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655216000:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758896.214434:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758896.214446:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758896/real 1787758896] req@ffff8c05a9f7b800 x1874588655216000/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758896.214455:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655216000/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758896.214461:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758896.214464:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758896.214466:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655216000:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758896.214486:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655216128:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758896.214510:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758896.214517:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758896/real 1787758896] req@ffff8c05a9f78e00 x1874588655216128/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758896.214525:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655216128/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758896.214532:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758896.214534:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758896.214536:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655216128:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758896.214540:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655216256:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758896.214552:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758896.214559:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758896/real 1787758896] req@ffff8c05a9f7ad80 x1874588655216256/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758896.214567:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655216256/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758896.214572:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758896.214574:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758896.214576:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655216256:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758896.214586:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655216384:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758896.214599:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758896.214606:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758896/real 1787758896] req@ffff8c05a9f7b100 x1874588655216384/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758896.214614:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655216384/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758896.214619:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758896.214621:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758896.214623:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655216384:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758896.214626:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655216512:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758896.214636:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758896.214642:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758896/real 1787758896] req@ffff8c05a9f79f80 x1874588655216512/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758896.214649:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655216512/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758896.214655:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758896.214657:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758896.214659:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655216512:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758896.214663:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655216640:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758896.214672:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758896.214677:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758896/real 1787758896] req@ffff8c05a9f7bb80 x1874588655216640/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758896.214691:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655216640/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758896.214697:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758896.214699:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758896.214701:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655216640:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758896.214704:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655216768:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758896.214715:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758896.214721:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758896/real 1787758896] req@ffff8c05a9f79180 x1874588655216768/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758896.214728:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655216768/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758896.214733:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758896.214735:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758896.214738:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655216768:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758896.342002:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758896.790131:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758897.366005:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758897.814063:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758898.390002:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758898.838100:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758899.413990:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758899.862090:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758900.437998:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758900.886097:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758901.334067:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758901.334076:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758901.334080:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758901.334090:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11957 00000100:00080000:1.0:1787758901.334101:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758901.334108:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758901.334148:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588655216896/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758901.334176:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758901.334179:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758901.334180:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758901.334183:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11957 00000100:00080000:1.0:1787758901.334185:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758901.334188:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758901.334196:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758901.334205:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655217024/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758901.334213:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758901.334215:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758901.334217:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758901.334219:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11957 00000100:00080000:1.0:1787758901.334222:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758901.334225:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758901.334228:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758901.334240:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655217152/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758901.334240:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655216896:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758901.334255:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758901.334258:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758901.334259:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758901.334261:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11957 00000100:00080000:1.0:1787758901.334263:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758901.334265:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758901.334268:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758901.334276:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655217280/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758901.334283:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758901.334285:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758901.334286:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758901.334288:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11957 00000100:00080000:1.0:1787758901.334290:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758901.334292:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758901.334295:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758901.334302:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655217408/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758901.334308:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758901.334311:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758901.334313:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758901.334315:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11957 00000100:00080000:1.0:1787758901.334318:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758901.334320:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758901.334323:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787758901.334329:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758901.334336:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655217536/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758901.334347:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758901.334351:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758901.334353:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758901.334357:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11957 00000100:00080000:1.0:1787758901.334361:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758901.334366:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758901.334372:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758901.334382:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758901/real 1787758901] req@ffff8c05b2d9a300 x1874588655216896/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758901.334387:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655217664/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758901.334412:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758901.334415:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758901.334420:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787758901.334421:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588655216896/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758901.334425:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11957 00000100:00080000:1.0:1787758901.334431:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758901.334438:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758901.334445:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787758901.334447:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758901.334455:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758901.334457:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655217792/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758901.334466:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655216896:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787758901.334477:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655217024:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758901.334504:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758901.334517:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758901/real 1787758901] req@ffff8c05b2d98380 x1874588655217024/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758901.334529:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655217024/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758901.334537:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758901.334540:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758901.334544:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655217024:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758901.334570:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655217152:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758901.334591:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758901.334602:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758901/real 1787758901] req@ffff8c05b2d99f80 x1874588655217152/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758901.334625:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655217152/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758901.334637:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758901.334640:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758901.334644:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655217152:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758901.334650:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655217280:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758901.334671:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758901.334684:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758901/real 1787758901] req@ffff8c05b2d9aa00 x1874588655217280/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758901.334695:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655217280/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758901.334703:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758901.334706:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758901.334710:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655217280:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758901.334716:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655217408:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758901.334735:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758901.334748:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758901/real 1787758901] req@ffff8c05b2d9b480 x1874588655217408/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758901.334760:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655217408/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758901.334769:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758901.334772:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758901.334775:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655217408:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758901.334780:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655217536:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758901.334800:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758901.334812:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758901/real 1787758901] req@ffff8c05b2d98700 x1874588655217536/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758901.334825:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655217536/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758901.334850:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758901.334854:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758901.334857:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655217536:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758901.334864:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655217664:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758901.334948:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758901.334965:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758901/real 1787758901] req@ffff8c05b2d98000 x1874588655217664/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758901.334977:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655217664/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758901.334986:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758901.334990:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758901.334993:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655217664:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758901.335000:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655217792:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758901.335020:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758901.335044:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758901/real 1787758901] req@ffff8c05b2d9b100 x1874588655217792/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758901.335056:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655217792/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758901.335065:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758901.335068:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758901.335071:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655217792:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758901.462008:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758901.910063:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758902.485950:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758902.934130:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758903.509955:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758903.958087:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758904.534039:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758904.982151:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758905.557979:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758906.006134:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758906.453964:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758906.453975:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758906.453981:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758906.453991:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11962 00000100:00080000:0.0:1787758906.454003:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758906.454011:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758906.454049:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655217920/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758906.454075:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758906.454078:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758906.454079:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758906.454083:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11962 00000100:00080000:0.0:1787758906.454087:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758906.454091:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758906.454100:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758906.454112:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655218048/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758906.454123:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758906.454128:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758906.454130:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787758906.454130:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655217920:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758906.454137:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11962 00000100:00080000:0.0:1787758906.454140:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758906.454144:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758906.454151:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758906.454160:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655218176/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758906.454171:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758906.454174:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758906.454176:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758906.454180:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11962 00000100:00080000:0.0:1787758906.454183:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758906.454188:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758906.454193:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758906.454204:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655218304/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758906.454213:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758906.454217:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758906.454219:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758906.454224:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11962 00000800:00000100:2.0:1787758906.454227:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758906.454229:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758906.454235:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758906.454240:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758906.454251:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655218432/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758906.454260:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758906.454263:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758906.454265:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758906.454269:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11962 00000100:00080000:0.0:1787758906.454273:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758906.454276:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758906.454281:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758906.454284:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758906/real 1787758906] req@ffff8c05a9f76680 x1874588655217920/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758906.454295:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588655218560/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758906.454306:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787758906.454310:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655217920/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758906.454311:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758906.454315:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758906.454322:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11962 00000100:00080000:0.0:1787758906.454328:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758906.454333:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:2.0:1787758906.454333:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787758906.454342:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787758906.454343:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787758906.454355:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588655218688/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758906.454361:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655217920:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758906.454371:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758906.454375:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758906.454377:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758906.454381:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11962 00000100:00080000:0.0:1787758906.454384:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758906.454387:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758906.454392:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758906.454395:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655218048:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758906.454404:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588655218816/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787758906.454424:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758906.454449:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758906/real 1787758906] req@ffff8c05a9f74380 x1874588655218048/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758906.454461:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655218048/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758906.454469:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758906.454472:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758906.454476:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655218048:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758906.454482:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655218176:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758906.454500:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758906.454510:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758906/real 1787758906] req@ffff8c05a9f77800 x1874588655218176/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758906.454521:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655218176/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758906.454528:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758906.454532:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758906.454535:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655218176:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758906.454541:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655218304:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758906.454559:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758906.454569:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758906/real 1787758906] req@ffff8c05a9f74a80 x1874588655218304/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758906.454581:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655218304/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758906.454589:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758906.454592:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758906.454595:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655218304:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758906.454601:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655218432:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758906.454620:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758906.454630:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758906/real 1787758906] req@ffff8c05a9f75880 x1874588655218432/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758906.454640:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655218432/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758906.454648:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758906.454651:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758906.454654:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655218432:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758906.454660:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655218560:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758906.454677:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758906.454686:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758906/real 1787758906] req@ffff8c05a9f75500 x1874588655218560/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758906.454697:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588655218560/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758906.454705:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758906.454708:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758906.454710:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655218560:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758906.454716:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655218688:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758906.454732:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758906.454763:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758906/real 1787758906] req@ffff8c05a9f74000 x1874588655218688/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758906.454774:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588655218688/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758906.454782:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758906.454785:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758906.454788:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655218688:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758906.454808:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655218816:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758906.454827:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758906.454837:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758906/real 1787758906] req@ffff8c05a9f75180 x1874588655218816/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758906.454847:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588655218816/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758906.454861:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758906.454864:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758906.454867:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655218816:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758906.581984:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758907.030139:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758907.605948:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758908.054000:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787758908.629947:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758909.078111:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758909.654001:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758910.102031:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758910.677996:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758911.126128:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787758911.574026:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758911.574038:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787758911.574043:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758911.574055:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11967 00000100:00080000:3.0:1787758911.574066:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758911.574074:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758911.574117:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655218944/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758911.574145:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758911.574149:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758911.574155:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758911.574159:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11967 00000100:00080000:3.0:1787758911.574162:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758911.574166:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758911.574177:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758911.574190:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655219072/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758911.574204:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655218944:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758911.574214:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758911.574219:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758911.574221:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758911.574226:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11967 00000100:00080000:3.0:1787758911.574229:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758911.574232:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758911.574237:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758911.574249:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655219200/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758911.574259:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758911.574263:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758911.574265:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758911.574269:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11967 00000100:00080000:3.0:1787758911.574272:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758911.574276:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758911.574282:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787758911.574283:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758911.574294:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655219328/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758911.574305:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758911.574309:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787758911.574311:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758911.574315:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11967 00000100:00080000:3.0:1787758911.574318:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758911.574322:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758911.574327:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758911.574336:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655219456/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758911.574345:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758911/real 1787758911] req@ffff8c05a9f79180 x1874588655218944/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758911.574347:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758911.574359:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787758911.574361:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758911.574366:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11967 00000100:00080000:3.0:1787758911.574370:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787758911.574372:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655218944/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758911.574376:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758911.574384:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787758911.574389:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787758911.574396:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655219584/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758911.574397:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758911.574406:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655218944:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787758911.574412:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758911.574417:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787758911.574419:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758911.574423:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11967 00000100:00080000:3.0:1787758911.574426:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758911.574430:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758911.574437:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758911.574441:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655219072:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758911.574448:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655219712/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758911.574459:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787758911.574465:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787758911.574468:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787758911.574471:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758911.574476:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11967 00000100:00080000:3.0:1787758911.574481:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787758911.574482:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758911/real 1787758911] req@ffff8c05a9f7bb80 x1874588655219072/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758911.574487:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787758911.574494:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758911.574504:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655219072/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758911.574507:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655219840/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758911.574538:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758911.574542:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758911.574545:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655219072:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758911.574551:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655219200:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758911.574566:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758911.574593:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758911/real 1787758911] req@ffff8c05a9f79f80 x1874588655219200/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758911.574605:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655219200/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758911.574613:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758911.574623:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758911.574626:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655219200:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758911.574632:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655219328:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758911.574666:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758911.574679:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758911/real 1787758911] req@ffff8c05a9f7b100 x1874588655219328/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758911.574692:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655219328/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758911.574699:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758911.574714:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758911.574719:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655219328:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758911.574725:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655219456:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758911.574744:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758911.574755:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758911/real 1787758911] req@ffff8c05a9f7ad80 x1874588655219456/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758911.574766:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655219456/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758911.574774:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758911.574777:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758911.574780:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655219456:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758911.574786:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655219584:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758911.574804:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758911.574820:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758911/real 1787758911] req@ffff8c05a9f78e00 x1874588655219584/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758911.574831:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655219584/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758911.574837:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758911.574839:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758911.574841:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655219584:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758911.574854:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655219712:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758911.574867:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758911.574908:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758911/real 1787758911] req@ffff8c05a9f7b800 x1874588655219712/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758911.574920:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655219712/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758911.574928:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758911.574931:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758911.574934:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655219712:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758911.574940:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655219840:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758911.574964:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758911.574974:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758911/real 1787758911] req@ffff8c05a9f79500 x1874588655219840/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758911.574985:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655219840/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758911.574990:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758911.574993:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758911.574995:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655219840:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758911.701966:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758912.150047:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758912.725985:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758913.174092:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758913.749939:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758914.198310:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758914.773986:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758915.222133:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758915.798026:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758916.246192:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758916.694022:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758916.694032:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758916.694037:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758916.694047:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11973 00000100:00080000:0.0:1787758916.694059:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758916.694066:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758916.694107:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588655219968/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758916.694137:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758916.694140:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758916.694141:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758916.694145:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11973 00000100:00080000:0.0:1787758916.694147:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758916.694151:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758916.694159:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758916.694172:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588655220096/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758916.694182:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758916.694185:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758916.694187:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758916.694190:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11973 00000100:00080000:0.0:1787758916.694194:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758916.694198:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758916.694203:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758916.694214:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588655220224/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758916.694225:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758916.694228:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758916.694230:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758916.694236:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11973 00000100:00080000:0.0:1787758916.694239:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758916.694244:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758916.694249:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758916.694263:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655219968:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787758916.694265:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588655220352/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758916.694289:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758916.694293:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758916.694294:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758916.694301:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11973 00000100:00080000:0.0:1787758916.694304:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758916.694308:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758916.694315:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758916.694330:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588655220480/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758916.694340:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758916.694344:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758916.694346:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787758916.694350:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758916.694355:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11973 00000100:00080000:0.0:1787758916.694362:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758916.694366:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758916.694376:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758916.694390:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588655220608/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758916.694400:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758916.694403:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758916.694405:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758916.694410:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11973 00000100:00100000:2.0:1787758916.694416:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758916/real 1787758916] req@ffff8c05a9f76d80 x1874588655219968/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758916.694419:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758916.694435:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758916.694444:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758916.694450:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588655219968/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758916.694463:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655220736/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758916.694464:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758916.694475:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758916.694486:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655219968:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758916.694498:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787758916.694504:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655220096:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758916.694506:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758916.694513:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758916.694522:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11973 00000100:00080000:0.0:1787758916.694527:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758916.694532:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:2.0:1787758916.694540:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:0.0:1787758916.694542:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758916.694555:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758916/real 1787758916] req@ffff8c05a9f77100 x1874588655220096/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758916.694573:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655220864/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758916.694597:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588655220096/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758916.694608:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758916.694612:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758916.694618:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655220096:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758916.694626:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655220224:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758916.694645:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758916.694655:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758916/real 1787758916] req@ffff8c05a9f74e00 x1874588655220224/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758916.694666:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588655220224/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758916.694674:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758916.694677:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758916.694680:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655220224:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758916.694708:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655220352:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758916.694738:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758916.694750:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758916/real 1787758916] req@ffff8c05a9f75180 x1874588655220352/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758916.694761:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588655220352/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758916.694768:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758916.694771:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758916.694774:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655220352:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758916.694779:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655220480:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758916.694796:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758916.694804:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758916/real 1787758916] req@ffff8c05a9f74000 x1874588655220480/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758916.694815:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588655220480/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758916.694821:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758916.694825:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758916.694828:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655220480:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758916.694833:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655220608:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758916.694855:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758916.694864:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758916/real 1787758916] req@ffff8c05a9f75500 x1874588655220608/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758916.694914:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588655220608/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758916.694922:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758916.694925:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758916.694928:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655220608:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758916.694934:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655220736:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758916.694951:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758916.694960:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758916/real 1787758916] req@ffff8c05a9f75880 x1874588655220736/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758916.694971:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655220736/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758916.694979:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758916.694982:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758916.694985:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655220736:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758916.694990:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655220864:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758916.695005:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758916.695013:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758916/real 1787758916] req@ffff8c05a9f74a80 x1874588655220864/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758916.695023:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655220864/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758916.695036:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758916.695039:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758916.695042:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655220864:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758916.822280:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758917.270153:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758917.846019:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758918.294105:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758918.870004:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758919.318151:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758919.893987:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758920.342160:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758920.917986:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758921.366154:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758921.814035:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758921.814047:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758921.814053:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758921.814067:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11978 00000100:00080000:0.0:1787758921.814100:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758921.814109:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758921.814154:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655220992/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758921.814183:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758921.814188:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758921.814191:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758921.814196:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11978 00000100:00080000:0.0:1787758921.814201:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758921.814205:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758921.814215:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758921.814226:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655220992:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787758921.814235:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655221120/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758921.814249:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758921.814253:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758921.814255:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758921.814260:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11978 00000100:00080000:0.0:1787758921.814263:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758921.814268:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758921.814273:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758921.814285:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655221248/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758921.814295:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758921.814299:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758921.814301:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758921.814305:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11978 00000100:00080000:0.0:1787758921.814311:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787758921.814314:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758921.814326:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758921.814333:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758921.814347:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655221376/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758921.814358:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758921.814361:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758921.814363:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758921.814370:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11978 00000100:00080000:0.0:1787758921.814374:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787758921.814381:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758921/real 1787758921] req@ffff8c05a9f77800 x1874588655220992/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758921.814382:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758921.814393:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758921.814416:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655220992/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758921.814418:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588655221504/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758921.814443:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758921.814449:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758921.814451:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758921.814453:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758921.814459:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11978 00000100:00080000:0.0:1787758921.814463:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758921.814465:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787758921.814484:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758921.814490:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758921.814506:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588655221632/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758921.814514:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758921.814518:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758921.814519:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758921.814523:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11978 00000100:00080000:0.0:1787758921.814526:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758921.814529:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758921.814534:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758921.814544:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588655221760/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758921.814552:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758921.814555:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758921.814557:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758921.814560:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11978 00000100:00080000:0.0:1787758921.814563:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758921.814567:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758921.814572:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758921.814581:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655221888/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758921.814639:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655220992:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787758921.814664:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655221120:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758921.814686:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758921.814697:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758921/real 1787758921] req@ffff8c05a9f74380 x1874588655221120/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758921.814708:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655221120/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758921.814715:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758921.814718:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758921.814721:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655221120:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758921.814727:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655221248:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758921.814741:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758921.814747:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758921/real 1787758921] req@ffff8c05a9f76680 x1874588655221248/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758921.814754:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655221248/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758921.814758:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758921.814760:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758921.814761:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655221248:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758921.814766:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655221376:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758921.814780:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758921.814788:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758921/real 1787758921] req@ffff8c05a9f76300 x1874588655221376/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758921.814796:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655221376/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758921.814803:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758921.814806:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758921.814809:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655221376:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758921.814813:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655221504:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758921.814823:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758921.814828:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758921/real 1787758921] req@ffff8c05a9f75c00 x1874588655221504/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758921.814835:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588655221504/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758921.814841:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758921.814843:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758921.814844:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655221504:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758921.814848:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655221632:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758921.814859:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758921.814864:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758921/real 1787758921] req@ffff8c05a9f76a00 x1874588655221632/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758921.814869:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588655221632/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758921.814904:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758921.814907:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758921.814909:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655221632:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758921.814913:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655221760:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758921.814931:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758921.814939:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758921/real 1787758921] req@ffff8c05a9f77b80 x1874588655221760/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758921.814946:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588655221760/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758921.814955:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758921.814957:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758921.814959:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655221760:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758921.814962:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655221888:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758921.814973:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758921.814978:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758921/real 1787758921] req@ffff8c05a9f74a80 x1874588655221888/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758921.814986:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655221888/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758921.814991:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758921.814993:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758921.814995:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655221888:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758921.942011:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758922.390132:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758922.965968:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758923.414136:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758923.989961:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758924.438141:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758925.014029:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758925.462173:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758926.037969:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758926.486030:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758926.934003:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758926.934017:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758926.934021:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758926.934032:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11983 00000100:00080000:0.0:1787758926.934042:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758926.934050:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758926.934093:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655222016/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758926.934125:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758926.934128:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758926.934130:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758926.934135:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11983 00000100:00080000:0.0:1787758926.934138:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758926.934142:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758926.934151:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758926.934167:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588655222144/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787758926.934178:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655222016:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758926.934179:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758926.934187:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758926.934190:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758926.934194:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11983 00000100:00080000:0.0:1787758926.934197:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758926.934201:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758926.934206:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758926.934220:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588655222272/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758926.934230:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758926.934234:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758926.934236:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758926.934239:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11983 00000100:00080000:0.0:1787758926.934243:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758926.934246:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:2.0:1787758926.934251:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:0.0:1787758926.934253:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758926.934265:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588655222400/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758926.934275:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758926.934278:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758926.934280:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758926.934284:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11983 00000100:00080000:0.0:1787758926.934288:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787758926.934292:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758926/real 1787758926] req@ffff8c05a9f75880 x1874588655222016/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758926.934297:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758926.934306:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758926.934314:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655222016/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758926.934321:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588655222528/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758926.934324:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758926.934330:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758926.934336:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655222016:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758926.934340:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758926.934344:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758926.934346:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758926.934351:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11983 00000100:00080000:0.0:1787758926.934354:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758926.934359:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758926.934363:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655222144:192.168.201.107@tcp:38:kworker.0 00000008:00100000:0.0:1787758926.934366:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758926.934395:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588655222656/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787758926.934404:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758926.934407:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758926.934411:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758926.934413:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787758926.934417:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758926/real 1787758926] req@ffff8c05a9f75500 x1874588655222144/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758926.934419:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11983 00000100:00080000:0.0:1787758926.934426:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758926.934433:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758926.934433:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588655222144/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787758926.934443:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787758926.934444:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758926.934450:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758926.934453:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655222144:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758926.934458:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655222272:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758926.934462:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588655222784/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758926.934473:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787758926.934476:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758926.934480:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758926.934483:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787758926.934487:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758926/real 1787758926] req@ffff8c05a9f74000 x1874588655222272/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758926.934490:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11983 00000100:00080000:0.0:1787758926.934497:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758926.934503:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758926.934508:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588655222272/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787758926.934509:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787758926.934522:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787758926.934527:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655222912/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758926.934528:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758926.934536:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655222272:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758926.934549:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655222400:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758926.934571:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758926.934582:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758926/real 1787758926] req@ffff8c05a9f75180 x1874588655222400/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758926.934594:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588655222400/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758926.934602:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758926.934605:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758926.934608:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655222400:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758926.934613:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655222528:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758926.934632:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758926.934640:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758926/real 1787758926] req@ffff8c05a9f74e00 x1874588655222528/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758926.934647:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588655222528/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758926.934652:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758926.934654:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758926.934656:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655222528:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758926.934667:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655222656:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787758926.934689:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758926.934699:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758926/real 1787758926] req@ffff8c05a9f77100 x1874588655222656/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758926.934710:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588655222656/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758926.934724:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758926.934727:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758926.934730:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655222656:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758926.934736:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655222784:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758926.934750:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758926.934757:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758926/real 1787758926] req@ffff8c05a9f76d80 x1874588655222784/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758926.934763:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588655222784/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758926.934768:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758926.934779:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758926.934782:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655222784:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787758926.934787:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655222912:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787758926.934805:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758926.934815:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758926/real 1787758926] req@ffff8c05a9f75880 x1874588655222912/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758926.934826:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655222912/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758926.934831:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787758926.934833:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787758926.934835:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655222912:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758927.061958:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758927.510137:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758928.085952:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758928.534110:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758929.109977:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758929.558066:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758930.134001:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758930.582076:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758931.157947:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758931.606201:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758932.054024:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758932.054034:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787758932.054037:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758932.054046:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11988 00000100:00080000:2.0:1787758932.054056:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758932.054062:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758932.054097:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588655223040/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758932.054130:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758932.054134:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758932.054136:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758932.054140:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11988 00000100:00080000:2.0:1787758932.054143:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758932.054147:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758932.054156:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758932.054171:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588655223168/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787758932.054175:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655223040:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787758932.054183:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758932.054186:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758932.054187:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758932.054190:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11988 00000100:00080000:2.0:1787758932.054192:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758932.054196:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758932.054199:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758932.054206:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588655223296/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758932.054213:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758932.054217:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758932.054218:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758932.054222:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11988 00000100:00080000:2.0:1787758932.054229:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758932.054233:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758932.054238:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758932.054249:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588655223424/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758932.054257:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758932.054260:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758932.054261:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758932.054263:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11988 00000100:00080000:2.0:1787758932.054266:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758932.054268:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758932.054271:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758932.054278:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588655223552/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758932.054282:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758932.054284:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758932.054286:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758932.054300:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11988 00000100:00080000:2.0:1787758932.054302:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758932.054304:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758932.054307:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758932.054314:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588655223680/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758932.054319:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758932.054321:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787758932.054322:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758932.054323:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11988 00000100:00080000:2.0:1787758932.054325:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758932.054327:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758932.054329:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758932.054335:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588655223808/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758932.054340:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758932.054343:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758932.054344:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758932.054346:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11988 00000100:00080000:2.0:1787758932.054347:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758932.054349:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758932.054353:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758932.054363:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588655223936/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787758932.054394:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655223168:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758932.054445:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655223296:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758932.054461:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655223424:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758932.054476:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655223552:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758932.054491:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655223680:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758932.054510:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655223808:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758932.054524:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655223936:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758932.182002:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758932.630104:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758933.205928:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758933.654124:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758934.229978:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758934.678136:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787758935.126128:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.107:988 00000400:00000100:0.0:1787758935.126174:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.107@tcp at host 192.168.201.107 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787758935.126192:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787758935.126293:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787758935.126302:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787758935.126312:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787758935.126323:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787758935.126334:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:3.0:1787758935.126348:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758932/real 1787758935] req@ffff8c05b2e73100 x1874588655223168/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787758935.126353:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787758935.126366:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:3.0:1787758935.126383:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588655223168/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758935.126397:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758935.126403:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758935.126406:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655223168:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758935.126414:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758932/real 1787758935] req@ffff8c05b2e70000 x1874588655223296/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758935.126424:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588655223296/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758935.126431:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758935.126434:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758935.126437:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655223296:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758935.126444:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758932/real 1787758935] req@ffff8c05b2e70a80 x1874588655223424/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758935.126454:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588655223424/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758935.126461:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758935.126465:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758935.126468:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655223424:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758935.126475:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758932/real 1787758935] req@ffff8c05b2e72300 x1874588655223552/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758935.126487:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588655223552/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758935.126494:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758935.126498:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758935.126501:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655223552:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758935.126508:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758932/real 1787758935] req@ffff8c05b2e72a00 x1874588655223680/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758935.126520:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588655223680/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758935.126527:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758935.126532:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758935.126534:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655223680:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758935.126541:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758932/real 1787758935] req@ffff8c05b2e72680 x1874588655223808/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758935.126551:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588655223808/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758935.126558:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758935.126561:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758935.126564:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655223808:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758935.126571:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758932/real 1787758935] req@ffff8c05b2e71180 x1874588655223936/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758935.126581:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588655223936/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758935.126588:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758935.126592:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758935.126594:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655223936:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758935.126601:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758932/real 1787758935] req@ffff8c05b2e72d80 x1874588655223040/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758935.126612:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588655223040/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758935.126619:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758935.126622:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758935.126624:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655223040:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787758935.254001:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758935.702128:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758936.278010:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758936.726050:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758937.174005:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758937.174016:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787758937.174020:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758937.174030:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11993 00000100:00080000:2.0:1787758937.174041:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758937.174047:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758937.174083:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588655224064/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758937.174114:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758937.174117:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758937.174119:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758937.174122:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11993 00000100:00080000:2.0:1787758937.174125:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758937.174128:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758937.174136:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758937.174146:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588655224192/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758937.174155:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758937.174158:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758937.174159:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758937.174162:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11993 00000100:00080000:2.0:1787758937.174165:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758937.174169:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758937.174172:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758937.174183:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588655224320/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758937.174191:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758937.174194:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758937.174196:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758937.174213:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11993 00000100:00080000:2.0:1787758937.174216:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758937.174223:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758937.174226:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655224064:192.168.201.107@tcp:250:kworker.0 00000008:00100000:2.0:1787758937.174230:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758937.174248:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588655224448/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758937.174259:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758937.174262:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758937.174263:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758937.174266:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11993 00000100:00080000:2.0:1787758937.174269:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758937.174273:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758937.174276:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758937.174285:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588655224576/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758937.174299:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758937.174301:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758937.174303:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758937.174306:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11993 00000100:00080000:2.0:1787758937.174308:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758937.174310:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758937.174313:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758937.174320:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588655224704/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758937.174327:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758937.174329:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787758937.174330:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758937.174333:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11993 00000100:00080000:2.0:1787758937.174335:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758937.174338:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758937.174350:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758937.174358:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588655224832/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758937.174366:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758937.174368:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758937.174370:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758937.174372:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11993 00000100:00080000:2.0:1787758937.174375:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758937.174379:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758937.174384:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758937.174392:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588655224960/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787758937.174464:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758937.174513:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758937/real 1787758937] req@ffff8c05b2e70380 x1874588655224064/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758937.174534:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588655224064/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758937.174545:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758937.174549:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758937.174553:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655224064:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787758937.174561:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655224192:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758937.174586:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758937.174599:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758937/real 1787758937] req@ffff8c05b2e70700 x1874588655224192/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758937.174612:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588655224192/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758937.174621:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758937.174624:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758937.174628:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655224192:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758937.174633:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655224320:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758937.175749:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758937.175771:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758937/real 1787758937] req@ffff8c05b2e73480 x1874588655224320/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758937.175786:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588655224320/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758937.175803:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758937.175807:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758937.175810:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655224320:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758937.175842:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655224448:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758937.175866:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758937.175893:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758937/real 1787758937] req@ffff8c05b2e71500 x1874588655224448/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758937.175905:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588655224448/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758937.175919:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758937.175922:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758937.175924:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655224448:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758937.175932:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655224576:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758937.175952:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758937.175962:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758937/real 1787758937] req@ffff8c05b2e73b80 x1874588655224576/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758937.175974:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588655224576/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758937.175981:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758937.175984:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758937.175988:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655224576:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758937.175994:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655224704:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758937.176015:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758937.176025:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758937/real 1787758937] req@ffff8c05b2e71c00 x1874588655224704/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758937.176035:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588655224704/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758937.176043:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758937.176046:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758937.176048:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655224704:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758937.176053:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655224832:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758937.176067:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758937.176074:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758937/real 1787758937] req@ffff8c05b2e71880 x1874588655224832/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758937.176081:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588655224832/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758937.176086:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758937.176088:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758937.176090:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655224832:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758937.176094:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655224960:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758937.176103:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758937.176112:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758937/real 1787758937] req@ffff8c05b2e70e00 x1874588655224960/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758937.176123:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588655224960/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758937.176130:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758937.176133:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758937.176137:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655224960:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758937.302010:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758937.750117:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758938.326032:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758938.774118:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758939.350000:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758939.798116:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758940.373985:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758940.822066:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758941.398010:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758941.846175:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758942.294054:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758942.294066:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758942.294070:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758942.294080:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 11998 00000100:00080000:0.0:1787758942.294091:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758942.294099:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758942.294159:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655225088/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758942.294189:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758942.294192:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758942.294194:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758942.294199:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11998 00000100:00080000:0.0:1787758942.294202:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758942.294206:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758942.294216:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758942.294230:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588655225216/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787758942.294235:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655225088:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758942.294245:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758942.294249:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758942.294251:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758942.294255:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11998 00000100:00080000:0.0:1787758942.294259:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758942.294262:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758942.294267:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758942.294279:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588655225344/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758942.294287:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758942.294291:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758942.294293:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758942.294296:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 11998 00000100:00080000:0.0:1787758942.294300:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787758942.294304:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758942.294308:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758942.294313:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758942.294324:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588655225472/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758942.294333:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758942.294336:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758942.294338:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758942.294342:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11998 00000100:00080000:0.0:1787758942.294345:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787758942.294351:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758942/real 1787758942] req@ffff8c05a9f74a80 x1874588655225088/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758942.294364:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758942.294369:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758942.294373:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655225088/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787758942.294383:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655225600/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758942.294388:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758942.294397:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787758942.294398:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787758942.294404:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1787758942.294408:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655225088:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758942.294409:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758942.294417:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11998 00000100:00080000:0.0:1787758942.294421:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758942.294424:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758942.294428:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758942.294438:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655225728/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787758942.294442:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655225216:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758942.294456:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758942.294461:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758942.294463:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758942.294467:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11998 00000100:00080000:0.0:1787758942.294471:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758942.294475:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758942.294480:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787758942.294480:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758942.294495:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655225856/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787758942.294496:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758942/real 1787758942] req@ffff8c05a9f77b80 x1874588655225216/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758942.294520:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787758942.294524:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588655225216/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758942.294526:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758942.294529:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758942.294536:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 11998 00000100:00080000:3.0:1787758942.294542:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758942.294543:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758942.294550:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:3.0:1787758942.294552:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000008:00100000:0.0:1787758942.294557:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758942.294558:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655225216:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758942.294568:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655225344:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758942.294571:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655225984/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787758942.294596:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758942.294609:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758942/real 1787758942] req@ffff8c05a9f76a00 x1874588655225344/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758942.294619:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588655225344/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758942.294624:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758942.294626:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758942.294628:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655225344:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758942.294631:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655225472:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758942.294644:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758942.294650:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758942/real 1787758942] req@ffff8c05a9f75c00 x1874588655225472/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758942.294661:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588655225472/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758942.294666:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758942.294667:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758942.294669:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655225472:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758942.294672:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655225600:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758942.294681:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758942.294687:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758942/real 1787758942] req@ffff8c05a9f76300 x1874588655225600/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758942.294693:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655225600/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758942.294697:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758942.294699:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758942.294700:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655225600:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758942.294711:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655225728:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758942.294721:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758942.294726:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758942/real 1787758942] req@ffff8c05a9f76680 x1874588655225728/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758942.294732:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655225728/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758942.294745:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758942.294748:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758942.294750:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655225728:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758942.294755:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655225856:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758942.294773:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758942.294781:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758942/real 1787758942] req@ffff8c05a9f74380 x1874588655225856/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758942.294790:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655225856/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758942.294797:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758942.294799:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758942.294802:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655225856:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758942.294808:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655225984:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758942.294824:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758942.294833:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758942/real 1787758942] req@ffff8c05a9f77800 x1874588655225984/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758942.294843:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655225984/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758942.294849:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758942.294852:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758942.294855:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655225984:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758942.422025:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758942.870137:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758943.446027:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758943.894133:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758944.470011:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758944.918082:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758945.493957:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758945.942123:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758946.517977:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758946.966039:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758947.414004:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758947.414030:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758947.414035:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758947.414045:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12003 00000100:00080000:0.0:1787758947.414057:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758947.414064:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758947.414114:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655226112/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758947.414146:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758947.414149:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758947.414151:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758947.414155:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12003 00000100:00080000:0.0:1787758947.414158:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758947.414162:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758947.414170:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758947.414183:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655226240/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758947.414193:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758947.414197:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758947.414198:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758947.414204:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12003 00000100:00100000:3.0:1787758947.414205:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655226112:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758947.414207:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758947.414214:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758947.414219:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758947.414229:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655226368/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758947.414237:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758947.414241:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758947.414242:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758947.414245:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12003 00000100:00080000:0.0:1787758947.414249:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758947.414252:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758947.414257:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758947.414265:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655226496/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758947.414274:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758947.414277:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758947.414281:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787758947.414282:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758947.414288:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12003 00000100:00080000:0.0:1787758947.414291:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758947.414295:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758947.414300:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758947.414310:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588655226624/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758947.414318:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758947.414321:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758947.414322:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758947.414326:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12003 00000100:00080000:0.0:1787758947.414329:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787758947.414331:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758947/real 1787758947] req@ffff8c05a9f77800 x1874588655226112/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758947.414335:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758947.414343:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758947.414354:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588655226752/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787758947.414358:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655226112/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758947.414372:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758947.414376:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758947.414378:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787758947.414379:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758947.414384:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12003 00000100:00080000:3.0:1787758947.414387:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787758947.414389:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787758947.414394:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655226112:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758947.414395:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758947.414404:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758947.414414:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588655226880/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758947.414421:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758947.414424:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758947.414426:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787758947.414428:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655226240:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758947.414432:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12003 00000100:00080000:0.0:1787758947.414437:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758947.414440:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758947.414444:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758947.414455:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655227008/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787758947.414464:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758947.414479:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758947/real 1787758947] req@ffff8c05a9f74380 x1874588655226240/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758947.414491:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655226240/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758947.414497:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758947.414500:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758947.414502:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655226240:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758947.414506:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655226368:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758947.414519:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758947.414525:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758947/real 1787758947] req@ffff8c05a9f76680 x1874588655226368/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758947.414532:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655226368/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758947.414536:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758947.414538:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758947.414540:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655226368:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758947.414543:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655226496:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758947.414552:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758947.414557:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758947/real 1787758947] req@ffff8c05a9f76300 x1874588655226496/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758947.414563:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655226496/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758947.414568:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758947.414569:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758947.414571:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655226496:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758947.414573:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655226624:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758947.414582:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758947.414587:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758947/real 1787758947] req@ffff8c05a9f75c00 x1874588655226624/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758947.414594:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588655226624/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758947.414598:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758947.414599:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758947.414601:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655226624:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758947.414604:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655226752:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758947.414612:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758947.414617:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758947/real 1787758947] req@ffff8c05a9f76a00 x1874588655226752/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758947.414622:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588655226752/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758947.414627:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758947.414629:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758947.414630:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655226752:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758947.414633:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655226880:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758947.414640:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758947.414647:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758947/real 1787758947] req@ffff8c05a9f77b80 x1874588655226880/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758947.414663:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588655226880/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758947.414671:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758947.414673:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758947.414677:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655226880:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758947.414695:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655227008:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758947.414709:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758947.414715:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758947/real 1787758947] req@ffff8c05a9f74a80 x1874588655227008/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758947.414722:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655227008/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758947.414726:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758947.414728:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758947.414730:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655227008:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758947.541946:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758947.990105:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758948.565958:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758949.014102:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758949.590035:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758950.038169:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758950.613977:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758951.062127:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758951.638027:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758952.086184:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758952.534059:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758952.534070:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758952.534080:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758952.534093:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12008 00000100:00080000:1.0:1787758952.534105:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758952.534112:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758952.534148:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655227136/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758952.534175:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758952.534178:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758952.534179:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758952.534182:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12008 00000100:00080000:1.0:1787758952.534185:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758952.534187:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758952.534195:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758952.534204:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655227264/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758952.534211:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758952.534214:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758952.534215:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758952.534219:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12008 00000100:00080000:1.0:1787758952.534221:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758952.534224:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758952.534227:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758952.534236:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655227392/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758952.534246:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787758952.534246:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655227136:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758952.534254:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758952.534257:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758952.534264:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12008 00000100:00080000:1.0:1787758952.534267:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758952.534270:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758952.534273:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758952.534282:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655227520/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758952.534288:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758952.534291:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758952.534292:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758952.534295:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12008 00000100:00080000:1.0:1787758952.534298:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758952.534300:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758952.534303:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758952.534310:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655227648/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758952.534316:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758952.534318:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758952.534318:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758952.534321:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12008 00000100:00080000:1.0:1787758952.534324:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758952.534326:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758952.534332:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787758952.534332:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758952.534343:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655227776/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758952.534358:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758952.534362:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758952.534364:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758952.534368:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12008 00000100:00080000:1.0:1787758952.534372:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758952.534377:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758952.534383:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758952.534387:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758952/real 1787758952] req@ffff8c05b2d9bb80 x1874588655227136/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758952.534400:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655227904/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787758952.534410:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655227136/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758952.534411:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758952.534417:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758952.534420:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758952.534428:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12008 00000100:00080000:1.0:1787758952.534433:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758952.534437:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:3.0:1787758952.534437:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787758952.534445:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787758952.534446:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758952.534454:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655227136:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787758952.534457:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655228032/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787758952.534468:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655227264:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758952.534496:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758952.534512:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758952/real 1787758952] req@ffff8c05b2d99880 x1874588655227264/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758952.534526:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655227264/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758952.534535:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758952.534538:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758952.534541:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655227264:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758952.534569:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655227392:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758952.534592:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758952.534605:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758952/real 1787758952] req@ffff8c05b2d9b100 x1874588655227392/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758952.534624:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655227392/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758952.534645:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758952.534649:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758952.534652:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655227392:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758952.534660:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655227520:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758952.534682:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758952.534695:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758952/real 1787758952] req@ffff8c05b2d98000 x1874588655227520/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758952.534708:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655227520/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758952.534717:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758952.534721:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758952.534724:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655227520:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758952.534731:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655227648:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758952.534751:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758952.534763:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758952/real 1787758952] req@ffff8c05b2d98700 x1874588655227648/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758952.534776:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655227648/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758952.534792:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758952.534795:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758952.534799:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655227648:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758952.534805:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655227776:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758952.534827:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758952.534838:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758952/real 1787758952] req@ffff8c05b2d9b480 x1874588655227776/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758952.534851:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655227776/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758952.534860:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758952.534863:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758952.534867:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655227776:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758952.534873:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655227904:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758952.534949:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758952.534964:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758952/real 1787758952] req@ffff8c05b2d9aa00 x1874588655227904/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758952.534977:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655227904/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758952.534986:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758952.534989:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758952.534993:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655227904:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758952.534999:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655228032:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758952.535021:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758952.535033:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758952/real 1787758952] req@ffff8c05b2d99f80 x1874588655228032/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758952.535045:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655228032/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758952.535053:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758952.535056:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758952.535060:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655228032:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758952.662053:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758953.110110:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758953.685999:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758954.134160:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758954.710050:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758955.158101:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758955.733968:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758956.182091:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758956.758024:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758957.206091:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758957.654016:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758957.654030:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758957.654035:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758957.654047:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12014 00000100:00080000:1.0:1787758957.654060:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758957.654068:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758957.654106:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655228160/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758957.654150:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758957.654154:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758957.654156:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758957.654160:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12014 00000100:00080000:1.0:1787758957.654164:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758957.654168:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758957.654178:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758957.654193:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588655228288/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758957.654206:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787758957.654211:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655228160:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758957.654212:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758957.654215:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758957.654225:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12014 00000100:00080000:1.0:1787758957.654229:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758957.654233:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758957.654239:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758957.654252:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655228416/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758957.654264:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758957.654268:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758957.654271:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758957.654275:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12014 00000100:00080000:1.0:1787758957.654281:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758957.654284:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758957.654290:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787758957.654297:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758957.654305:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655228544/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758957.654317:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758957.654321:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758957.654323:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758957.654327:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12014 00000100:00080000:1.0:1787758957.654336:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758957.654340:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758957.654345:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758957.654354:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655228672/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758957.654364:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758957.654368:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758957.654370:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758957.654374:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12014 00000100:00080000:1.0:1787758957.654377:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758957.654381:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758957.654386:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758957.654397:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655228800/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758957.654408:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758957.654412:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758957.654416:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787758957.654416:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758957/real 1787758957] req@ffff8c05b2d98380 x1874588655228160/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758957.654425:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12014 00000100:00080000:1.0:1787758957.654432:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758957.654437:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758957.654442:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758957.654443:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655228160/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758957.654455:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787758957.654457:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655228928/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758957.654469:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758957.654473:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758957.654475:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758957.654478:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12014 00000100:00080000:1.0:1787758957.654481:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758957.654484:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758957.654488:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758957.654496:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655229056/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758957.654584:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758957.654589:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655228160:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787758957.654614:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655228288:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758957.654671:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758957.654688:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758957/real 1787758957] req@ffff8c05b2d9a300 x1874588655228288/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758957.654707:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588655228288/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758957.654717:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758957.654721:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758957.654725:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655228288:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758957.654732:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655228416:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758957.654753:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758957.654766:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758957/real 1787758957] req@ffff8c05b2d99f80 x1874588655228416/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758957.654778:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655228416/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758957.654786:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758957.654788:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758957.654791:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655228416:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758957.654795:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655228544:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758957.654819:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758957.654830:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758957/real 1787758957] req@ffff8c05b2d9aa00 x1874588655228544/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758957.654857:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655228544/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758957.654864:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758957.654866:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758957.654868:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655228544:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758957.654873:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655228672:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758957.654912:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758957.654925:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758957/real 1787758957] req@ffff8c05b2d9b480 x1874588655228672/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758957.654933:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655228672/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758957.654938:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758957.654940:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758957.654942:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655228672:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758957.654946:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655228800:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758957.654957:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758957.654963:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758957/real 1787758957] req@ffff8c05b2d98700 x1874588655228800/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758957.654976:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655228800/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758957.654981:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758957.654983:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758957.654984:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655228800:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758957.654988:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655228928:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758957.654998:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758957.655003:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758957/real 1787758957] req@ffff8c05b2d98000 x1874588655228928/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758957.655011:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655228928/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758957.655016:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758957.655018:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758957.655020:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655228928:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758957.655023:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655229056:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758957.655031:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758957.655036:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758957/real 1787758957] req@ffff8c05b2d9b100 x1874588655229056/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758957.655043:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655229056/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758957.655052:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758957.655055:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758957.655057:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655229056:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758957.782041:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758958.230151:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758958.806028:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758959.254122:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758959.830019:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758960.278189:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758960.854003:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758961.302052:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758961.878006:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758962.326159:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787758962.774071:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758962.774085:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787758962.774090:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758962.774103:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12019 00000100:00080000:1.0:1787758962.774116:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758962.774125:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758962.774169:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655229184/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758962.774208:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758962.774213:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758962.774215:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758962.774219:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12019 00000100:00080000:1.0:1787758962.774223:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758962.774228:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758962.774238:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758962.774254:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655229312/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787758962.774261:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655229184:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758962.774273:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758962.774278:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758962.774280:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758962.774286:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12019 00000100:00080000:1.0:1787758962.774290:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758962.774295:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758962.774301:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758962.774317:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655229440/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758962.774330:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787758962.774332:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758962.774337:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758962.774340:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758962.774346:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12019 00000100:00080000:1.0:1787758962.774357:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758962.774362:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758962.774368:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758962.774381:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655229568/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758962.774391:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787758962.774395:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758962/real 1787758962] req@ffff8c05b2d99880 x1874588655229184/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758962.774397:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787758962.774401:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758962.774410:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12019 00000100:00080000:1.0:1787758962.774416:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758962.774423:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758962.774428:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758962.774430:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655229184/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758962.774443:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655229696/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758962.774446:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758962.774452:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758962.774459:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655229184:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787758962.774470:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787758962.774475:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787758962.774477:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758962.774483:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12019 00000100:00080000:1.0:1787758962.774486:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758962.774491:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758962.774496:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655229312:192.168.201.107@tcp:38:kworker.0 00000008:00100000:1.0:1787758962.774499:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758962.774516:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655229824/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787758962.774527:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787758962.774533:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787758962.774535:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787758962.774539:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758962.774544:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12019 00000100:00080000:1.0:1787758962.774548:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787758962.774549:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758962/real 1787758962] req@ffff8c05b2d9bb80 x1874588655229312/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758962.774564:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758962.774570:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655229312/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787758962.774571:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787758962.774585:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787758962.774590:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655229952/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758962.774592:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758962.774602:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655229312:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758962.774611:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787758962.774616:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655229440:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787758962.774618:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787758962.774622:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787758962.774629:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12019 00000100:00080000:1.0:1787758962.774634:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758962.774639:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787758962.774645:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787758962.774646:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758962.774659:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758962/real 1787758962] req@ffff8c05b2d9b100 x1874588655229440/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758962.774663:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655230080/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787758962.774686:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655229440/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758962.774709:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758962.774718:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758962.774722:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655229440:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758962.774729:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655229568:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758962.774834:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758962.774848:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758962/real 1787758962] req@ffff8c05b2d98000 x1874588655229568/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758962.774862:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655229568/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758962.774910:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758962.774915:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758962.774919:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655229568:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758962.774926:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655229696:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758962.774948:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758962.774961:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758962/real 1787758962] req@ffff8c05b2d98700 x1874588655229696/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758962.774974:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655229696/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758962.774984:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758962.774988:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758962.774991:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655229696:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758962.775012:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655229824:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758962.775035:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758962.775048:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758962/real 1787758962] req@ffff8c05b2d9b480 x1874588655229824/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758962.775061:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655229824/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758962.775070:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758962.775073:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758962.775078:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655229824:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758962.775084:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655229952:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758962.775107:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758962.775120:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758962/real 1787758962] req@ffff8c05b2d9aa00 x1874588655229952/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758962.775133:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655229952/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758962.775142:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758962.775145:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758962.775149:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655229952:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758962.775155:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655230080:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758962.775176:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758962.775187:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758962/real 1787758962] req@ffff8c05b2d99f80 x1874588655230080/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758962.775200:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655230080/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758962.775209:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758962.775213:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758962.775216:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655230080:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758962.901987:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758963.350054:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758963.925987:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758964.374176:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758964.949990:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758965.398094:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758965.973995:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758966.422182:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758966.997969:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758967.446145:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758967.893996:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758967.894005:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787758967.894009:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758967.894019:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12024 00000100:00080000:2.0:1787758967.894030:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758967.894038:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758967.894094:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73800 x1874588655230208/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758967.894124:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758967.894127:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758967.894129:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758967.894133:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12024 00000100:00080000:2.0:1787758967.894138:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758967.894142:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758967.894152:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758967.894168:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588655230336/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758967.894179:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758967.894183:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758967.894185:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758967.894189:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12024 00000100:00080000:2.0:1787758967.894194:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787758967.894196:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655230208:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787758967.894202:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758967.894210:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758967.894223:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588655230464/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758967.894241:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758967.894244:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758967.894246:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758967.894252:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12024 00000100:00080000:2.0:1787758967.894261:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758967.894264:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758967.894268:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787758967.894276:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758967.894279:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588655230592/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758967.894288:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758967.894290:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758967.894291:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758967.894294:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12024 00000100:00080000:2.0:1787758967.894297:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758967.894300:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758967.894304:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758967.894312:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588655230720/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758967.894318:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758967.894321:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758967.894323:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758967.894325:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12024 00000100:00080000:2.0:1787758967.894327:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758967.894329:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758967.894332:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758967.894340:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588655230848/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787758967.894340:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758967/real 1787758967] req@ffff8c05b2e73800 x1874588655230208/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758967.894361:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758967.894365:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787758967.894367:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758967.894370:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12024 00000100:00080000:2.0:1787758967.894373:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787758967.894376:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73800 x1874588655230208/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758967.894378:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758967.894384:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787758967.894394:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787758967.894395:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588655230976/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758967.894400:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758967.894408:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655230208:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787758967.894410:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758967.894416:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758967.894418:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787758967.894419:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655230336:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787758967.894425:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12024 00000100:00080000:2.0:1787758967.894430:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758967.894434:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758967.894439:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787758967.894445:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758967.894454:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588655231104/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787758967.894459:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758967/real 1787758967] req@ffff8c05b2e70e00 x1874588655230336/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758967.894478:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588655230336/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758967.894485:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758967.894488:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758967.894490:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655230336:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758967.894515:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655230464:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758967.894527:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758967.894534:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758967/real 1787758967] req@ffff8c05b2e71880 x1874588655230464/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758967.894542:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588655230464/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758967.894548:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758967.894550:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758967.894552:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655230464:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758967.894556:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655230592:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758967.894566:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758967.894572:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758967/real 1787758967] req@ffff8c05b2e71c00 x1874588655230592/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758967.894579:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588655230592/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758967.894585:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758967.894586:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758967.894588:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655230592:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758967.894592:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655230720:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758967.894603:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758967.894608:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758967/real 1787758967] req@ffff8c05b2e73b80 x1874588655230720/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758967.894616:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588655230720/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758967.894621:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758967.894623:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758967.894625:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655230720:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758967.894628:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655230848:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758967.894637:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758967.894643:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758967/real 1787758967] req@ffff8c05b2e71500 x1874588655230848/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758967.894679:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588655230848/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758967.894688:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758967.894703:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758967.894711:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655230848:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758967.894715:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655230976:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758967.894732:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758967.894740:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758967/real 1787758967] req@ffff8c05b2e73480 x1874588655230976/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758967.894747:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588655230976/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758967.894758:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758967.894760:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758967.894762:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655230976:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758967.894766:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655231104:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758967.894777:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758967.894782:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758967/real 1787758967] req@ffff8c05b2e70700 x1874588655231104/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758967.894789:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588655231104/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758967.894795:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758967.894796:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758967.894798:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655231104:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758968.021951:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758968.470105:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758969.045991:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758969.494005:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758970.069995:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758970.518095:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758971.094008:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758971.542082:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758972.118007:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758972.566106:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758973.014003:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758973.014015:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787758973.014021:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758973.014032:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12029 00000100:00080000:2.0:1787758973.014044:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758973.014052:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758973.014095:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588655231232/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758973.014121:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758973.014124:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758973.014125:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758973.014129:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12029 00000100:00080000:2.0:1787758973.014132:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758973.014135:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758973.014143:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758973.014152:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588655231360/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758973.014165:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758973.014168:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758973.014170:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758973.014174:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12029 00000100:00080000:2.0:1787758973.014176:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758973.014178:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758973.014181:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758973.014188:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588655231488/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758973.014203:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758973.014207:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758973.014209:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758973.014215:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12029 00000100:00080000:2.0:1787758973.014223:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787758973.014224:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655231232:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787758973.014234:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758973.014242:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758973.014258:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588655231616/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758973.014266:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758973.014269:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758973.014270:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758973.014273:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12029 00000100:00080000:2.0:1787758973.014276:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758973.014288:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758973.014292:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758973.014301:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588655231744/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758973.014308:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758973.014310:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758973.014312:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758973.014314:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12029 00000100:00080000:2.0:1787758973.014317:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758973.014319:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758973.014322:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787758973.014330:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787758973.014336:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588655231872/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758973.014345:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758973.014349:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787758973.014350:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758973.014354:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12029 00000100:00080000:2.0:1787758973.014358:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758973.014360:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758973.014363:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758973.014370:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588655232000/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758973.014381:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758973.014384:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758973.014386:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758973.014389:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12029 00000100:00080000:2.0:1787758973.014390:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758973.014398:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758973.014399:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758973/real 1787758973] req@ffff8c05b2e70380 x1874588655231232/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787758973.014402:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758973.014419:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588655232128/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787758973.014474:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588655231232/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787758998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758973.014488:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758973.014493:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758973.014498:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655231232:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787758973.014506:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655231360:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758973.014535:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758973.014548:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758973/real 1787758973] req@ffff8c05b2e72d80 x1874588655231360/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758973.014560:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588655231360/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758973.014582:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758973.014585:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758973.014589:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655231360:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758973.014617:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655231488:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758973.014638:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758973.014646:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758973/real 1787758973] req@ffff8c05b2e71180 x1874588655231488/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758973.014654:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588655231488/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758973.014670:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758973.014673:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758973.014675:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655231488:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758973.014679:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655231616:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758973.014693:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758973.014699:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758973/real 1787758973] req@ffff8c05b2e72680 x1874588655231616/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758973.014705:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588655231616/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758973.014710:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758973.014712:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758973.014714:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655231616:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758973.014717:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655231744:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758973.014727:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758973.014732:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758973/real 1787758973] req@ffff8c05b2e72a00 x1874588655231744/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758973.014738:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588655231744/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758973.014743:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758973.014744:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758973.014747:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655231744:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758973.014750:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655231872:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758973.014759:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758973.014764:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758973/real 1787758973] req@ffff8c05b2e72300 x1874588655231872/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758973.014770:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588655231872/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787758998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758973.014774:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758973.014776:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758973.014777:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655231872:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758973.014781:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655232000:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758973.014793:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758973.014802:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758973/real 1787758973] req@ffff8c05b2e70a80 x1874588655232000/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758973.014813:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588655232000/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758973.014821:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758973.014823:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758973.014826:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655232000:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758973.014833:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655232128:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758973.014852:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758973.014873:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758973/real 1787758973] req@ffff8c05b2e70000 x1874588655232128/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758973.014923:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588655232128/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787758998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758973.014928:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758973.014930:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758973.014932:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655232128:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758973.142039:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758973.590080:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758974.165989:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758974.614110:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758975.189951:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758975.638047:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758976.213992:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758976.662088:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758977.238005:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758977.686081:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758978.133986:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758978.133994:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787758978.133998:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758978.134008:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12034 00000100:00080000:2.0:1787758978.134019:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758978.134025:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758978.134060:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588655232256/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758978.134082:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758978.134085:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758978.134086:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758978.134089:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12034 00000100:00080000:2.0:1787758978.134091:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758978.134093:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758978.134101:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758978.134111:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588655232384/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758978.134117:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758978.134120:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758978.134121:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758978.134123:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12034 00000100:00080000:2.0:1787758978.134125:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758978.134128:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758978.134130:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655232256:192.168.201.107@tcp:250:kworker.0 00000008:00100000:2.0:1787758978.134131:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758978.134141:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588655232512/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758978.134147:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758978.134150:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758978.134151:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758978.134153:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12034 00000100:00080000:2.0:1787758978.134155:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758978.134157:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758978.134159:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758978.134167:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588655232640/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758978.134173:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758978.134175:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758978.134177:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758978.134179:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12034 00000100:00080000:2.0:1787758978.134182:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758978.134184:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758978.134187:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758978.134193:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588655232768/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758978.134199:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787758978.134200:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787758978.134201:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758978.134204:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758978.134208:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12034 00000100:00080000:2.0:1787758978.134211:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758978.134213:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758978.134215:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758978.134221:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588655232896/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758978.134227:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758978.134228:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787758978.134229:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758978.134231:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12034 00000100:00080000:2.0:1787758978.134233:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758978.134235:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758978.134239:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758978.134245:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588655233024/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758978.134251:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758978.134253:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758978.134255:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787758978.134258:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758978/real 1787758978] req@ffff8c05b2e73100 x1874588655232256/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758978.134259:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12034 00000100:00080000:2.0:1787758978.134263:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758978.134267:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758978.134270:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758978.134280:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588655233152/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787758978.134298:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588655232256/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758978.134308:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758978.134311:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758978.134314:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655232256:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787758978.134321:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655232384:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758978.134340:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758978.134349:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758978/real 1787758978] req@ffff8c05b2e70000 x1874588655232384/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758978.134357:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588655232384/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758978.134363:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758978.134365:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758978.134367:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655232384:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758978.134385:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655232512:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758978.134394:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758978.134399:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758978/real 1787758978] req@ffff8c05b2e70a80 x1874588655232512/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758978.134406:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588655232512/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758978.134411:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758978.134414:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758978.134416:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655232512:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758978.134419:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655232640:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758978.134428:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758978.134433:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758978/real 1787758978] req@ffff8c05b2e72300 x1874588655232640/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758978.134440:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588655232640/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758978.134444:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758978.134446:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758978.134448:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655232640:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758978.134451:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655232768:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758978.134460:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758978.134465:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758978/real 1787758978] req@ffff8c05b2e72a00 x1874588655232768/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758978.134471:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588655232768/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758978.134493:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758978.134496:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758978.134498:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655232768:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758978.134502:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655232896:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758978.134516:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758978.134522:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758978/real 1787758978] req@ffff8c05b2e72680 x1874588655232896/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758978.134529:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588655232896/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758978.134538:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758978.134540:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758978.134542:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655232896:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758978.134544:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655233024:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758978.134553:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758978.134558:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758978/real 1787758978] req@ffff8c05b2e71180 x1874588655233024/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758978.134564:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588655233024/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758978.134569:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758978.134570:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758978.134572:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655233024:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758978.134575:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655233152:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758978.134584:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758978.134588:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758978/real 1787758978] req@ffff8c05b2e72d80 x1874588655233152/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758978.134595:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588655233152/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758978.134599:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758978.134600:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758978.134603:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655233152:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758978.261985:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758978.710132:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758979.285951:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758979.734028:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758980.310008:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758980.758098:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758981.333979:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758981.782058:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758982.357967:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758982.806163:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787758983.253981:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758983.253993:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787758983.253996:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758983.254007:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12039 00000100:00080000:2.0:1787758983.254017:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758983.254026:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787758983.254070:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588655233280/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758983.254099:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758983.254102:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758983.254104:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758983.254109:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12039 00000100:00080000:2.0:1787758983.254112:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758983.254117:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758983.254126:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758983.254141:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588655233408/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787758983.254142:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655233280:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787758983.254158:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758983.254161:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758983.254162:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758983.254165:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12039 00000100:00080000:2.0:1787758983.254168:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758983.254170:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758983.254173:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758983.254181:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588655233536/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758983.254189:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758983.254192:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758983.254193:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758983.254196:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12039 00000100:00080000:2.0:1787758983.254208:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758983.254211:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758983.254214:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787758983.254222:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588655233664/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787758983.254228:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787758983.254233:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758983.254236:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787758983.254237:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758983.254240:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12039 00000100:00080000:2.0:1787758983.254243:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758983.254246:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758983.254249:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787758983.254258:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588655233792/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758983.254263:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758983.254265:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787758983.254267:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758983.254270:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12039 00000100:00080000:2.0:1787758983.254272:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758983.254275:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758983.254278:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758983.254286:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758983/real 1787758983] req@ffff8c05b2e70380 x1874588655233280/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787758983.254288:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588655233920/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787758983.254318:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758983.254321:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787758983.254323:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758983.254326:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12039 00000100:00080000:2.0:1787758983.254329:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787758983.254330:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588655233280/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787758983.254334:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758983.254342:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787758983.254346:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787758983.254351:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588655234048/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787758983.254352:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758983.254360:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655233280:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787758983.254369:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787758983.254371:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787758983.254373:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787758983.254376:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12039 00000100:00080000:2.0:1787758983.254378:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787758983.254381:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787758983.254383:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787758983.254406:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655233408:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787758983.254420:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588655234176/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787758983.254439:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758983.254455:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758983/real 1787758983] req@ffff8c05b2e70700 x1874588655233408/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758983.254465:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588655233408/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758983.254472:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758983.254475:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758983.254477:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655233408:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758983.254481:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655233536:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758983.254495:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758983.254502:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758983/real 1787758983] req@ffff8c05b2e73480 x1874588655233536/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758983.254510:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588655233536/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758983.254515:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758983.254518:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758983.254520:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655233536:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758983.254523:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655233664:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758983.254534:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758983.254539:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758983/real 1787758983] req@ffff8c05b2e71500 x1874588655233664/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758983.254547:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588655233664/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758983.254552:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758983.254554:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758983.254555:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655233664:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758983.254559:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655233792:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758983.254570:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758983.254576:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758983/real 1787758983] req@ffff8c05b2e73b80 x1874588655233792/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758983.254584:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588655233792/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758983.254589:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758983.254591:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758983.254593:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655233792:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758983.254596:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655233920:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758983.254606:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758983.254612:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758983/real 1787758983] req@ffff8c05b2e71c00 x1874588655233920/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758983.254619:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588655233920/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758983.254624:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758983.254626:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758983.254628:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655233920:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758983.254631:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655234048:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758983.254652:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758983.254659:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758983/real 1787758983] req@ffff8c05b2e71880 x1874588655234048/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758983.254676:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588655234048/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758983.254681:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758983.254683:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758983.254685:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655234048:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758983.254699:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655234176:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758983.254711:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758983.254716:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758983/real 1787758983] req@ffff8c05b2e70e00 x1874588655234176/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758983.254724:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588655234176/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758983.254729:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758983.254732:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758983.254736:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655234176:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758983.381969:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758983.830074:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758984.405983:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758984.854105:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758985.430035:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758985.878107:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758986.454002:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758986.902126:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758987.477986:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758987.926063:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758988.374036:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758988.374051:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758988.374056:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758988.374068:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12044 00000100:00080000:0.0:1787758988.374081:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758988.374091:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758988.374137:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655234304/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758988.374172:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758988.374176:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758988.374179:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758988.374183:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12044 00000100:00080000:0.0:1787758988.374188:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758988.374193:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758988.374203:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758988.374221:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588655234432/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758988.374233:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758988.374238:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758988.374240:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758988.374245:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12044 00000100:00080000:0.0:1787758988.374249:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758988.374262:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787758988.374264:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655234304:192.168.201.107@tcp:250:kworker.0 00000008:00100000:0.0:1787758988.374272:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758988.374294:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588655234560/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758988.374326:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758988.374330:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758988.374334:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758988.374339:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12044 00000100:00080000:0.0:1787758988.374343:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758988.374349:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758988.374355:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758988.374371:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588655234688/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787758988.374375:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758988.374390:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758988.374395:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758988.374398:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758988.374404:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12044 00000100:00080000:0.0:1787758988.374409:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758988.374416:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758988.374424:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758988.374438:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588655234816/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787758988.374442:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758988/real 1787758988] req@ffff8c05a9f75880 x1874588655234304/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758988.374485:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758988.374490:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1787758988.374494:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655234304/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758988.374495:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758988.374514:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12044 00000100:00080000:3.0:1787758988.374517:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758988.374521:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787758988.374523:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758988.374533:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655234304:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758988.374536:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758988.374554:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787758988.374559:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655234432:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758988.374580:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588655234944/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787758988.374589:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758988.374604:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787758988.374604:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758988/real 1787758988] req@ffff8c05a9f76d80 x1874588655234432/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758988.374621:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758988.374629:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787758988.374631:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588655234432/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758988.374639:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12044 00000100:00080000:0.0:1787758988.374647:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758988.374657:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:3.0:1787758988.374657:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787758988.374671:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787758988.374672:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758988.374682:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655234432:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758988.374693:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588655235072/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758988.374705:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758988.374710:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758988.374713:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787758988.374719:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655234560:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787758988.374720:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12044 00000100:00080000:0.0:1787758988.374730:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758988.374738:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758988.374745:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787758988.374759:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787758988.374765:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588655235200/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787758988.374779:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758988/real 1787758988] req@ffff8c05a9f77100 x1874588655234560/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758988.374808:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588655234560/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758988.374818:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758988.374822:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758988.374826:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655234560:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758988.374834:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655234688:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758988.374858:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758988.374931:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758988/real 1787758988] req@ffff8c05a9f74e00 x1874588655234688/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758988.374945:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588655234688/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758988.374955:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758988.374970:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758988.374974:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655234688:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758988.374982:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655234816:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758988.375006:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758988.375019:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758988/real 1787758988] req@ffff8c05a9f75180 x1874588655234816/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758988.375032:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588655234816/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758988.375049:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758988.375052:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758988.375056:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655234816:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758988.375063:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655234944:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787758988.375084:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758988.375097:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758988/real 1787758988] req@ffff8c05a9f74000 x1874588655234944/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758988.375120:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588655234944/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758988.375129:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758988.375133:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758988.375137:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655234944:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787758988.375144:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655235072:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758988.375165:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758988.375176:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758988/real 1787758988] req@ffff8c05a9f75500 x1874588655235072/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758988.375189:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588655235072/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758988.375198:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758988.375201:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758988.375205:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655235072:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758988.375227:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655235200:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787758988.375250:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787758988.375262:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758988/real 1787758988] req@ffff8c05a9f76d80 x1874588655235200/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787758988.375274:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588655235200/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787758988.375282:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787758988.375286:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787758988.375289:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655235200:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758988.502029:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758988.950184:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758989.526013:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758989.974144:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758990.549970:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758990.998150:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758991.574028:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758992.022143:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758992.598044:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758993.046118:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758993.493931:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758993.493945:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758993.493949:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758993.493969:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12049 00000100:00080000:0.0:1787758993.493981:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758993.493988:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758993.494031:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655235328/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758993.494061:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758993.494065:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758993.494066:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758993.494071:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12049 00000100:00080000:0.0:1787758993.494074:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758993.494077:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758993.494093:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758993.494108:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655235456/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758993.494116:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655235328:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758993.494127:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758993.494130:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758993.494132:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758993.494137:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12049 00000100:00080000:0.0:1787758993.494141:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758993.494145:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758993.494150:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758993.494162:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588655235584/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758993.494171:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758993.494175:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758993.494176:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758993.494179:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12049 00000100:00080000:0.0:1787758993.494183:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758993.494187:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758993.494197:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758993.494223:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588655235712/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758993.494233:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758993.494236:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758993.494238:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758993.494241:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12049 00000100:00080000:0.0:1787758993.494246:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758993.494250:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758993.494256:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758993.494277:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588655235840/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758993.494288:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758993.494297:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:1.0:1787758993.494298:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787758993.494300:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758993.494308:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12049 00000100:00080000:0.0:1787758993.494312:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758993.494316:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758993.494322:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758993.494337:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655235968/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758993.494349:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758993.494352:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758993.494354:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758993.494357:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12049 00000100:00080000:0.0:1787758993.494362:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758993.494365:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758993.494371:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758993.494386:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655236096/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758993.494387:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758993/real 1787758993] req@ffff8c05a9f75880 x1874588655235328/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758993.494440:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758993.494446:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758993.494448:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787758993.494456:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655235328/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787758993.494460:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12049 00000100:00080000:1.0:1787758993.494475:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787758993.494477:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787758993.494482:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787758993.494489:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787758993.494489:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655235328:192.168.201.107@tcp:250:kworker.0 00000008:00100000:0.0:1787758993.494506:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758993.494520:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655236224/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758993.494540:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655235456:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758993.494595:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758993.494623:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758993/real 1787758993] req@ffff8c05a9f74a80 x1874588655235456/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758993.494646:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655235456/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758993.494657:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758993.494663:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758993.494669:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655235456:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758993.494679:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655235584:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758993.494711:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758993.494725:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758993/real 1787758993] req@ffff8c05a9f77b80 x1874588655235584/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758993.494739:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588655235584/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758993.494748:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758993.494752:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758993.494754:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655235584:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758993.494760:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655235712:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758993.494778:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758993.494785:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758993/real 1787758993] req@ffff8c05a9f76a00 x1874588655235712/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758993.494793:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588655235712/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758993.494803:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758993.494806:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758993.494808:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655235712:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758993.494812:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655235840:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758993.494824:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758993.494830:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758993/real 1787758993] req@ffff8c05a9f75c00 x1874588655235840/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758993.494837:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588655235840/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758993.494842:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758993.494844:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758993.494846:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655235840:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758993.494850:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655235968:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787758993.494860:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758993.494866:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758993/real 1787758993] req@ffff8c05a9f76300 x1874588655235968/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758993.494873:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655235968/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758993.494922:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758993.494926:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758993.494932:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655235968:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758993.494940:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655236096:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758993.494958:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758993.494965:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758993/real 1787758993] req@ffff8c05a9f76680 x1874588655236096/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758993.494973:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655236096/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758993.494978:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758993.494980:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758993.494983:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655236096:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758993.494986:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655236224:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787758993.494997:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787758993.495002:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758993/real 1787758993] req@ffff8c05a9f74380 x1874588655236224/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787758993.495009:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655236224/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787758993.495016:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787758993.495019:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787758993.495022:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655236224:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758993.622007:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758994.070468:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758994.646023:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758995.094084:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758995.669991:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758996.118048:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758996.693977:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787758997.142125:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758997.718004:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787758998.166049:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787758998.613984:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758998.613996:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787758998.614001:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758998.614012:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12055 00000100:00080000:0.0:1787758998.614021:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758998.614038:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787758998.614089:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655236352/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758998.614130:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758998.614132:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758998.614134:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758998.614138:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12055 00000100:00080000:0.0:1787758998.614141:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758998.614144:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758998.614153:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758998.614166:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655236480/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758998.614175:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758998.614176:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655236352:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787758998.614178:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758998.614181:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758998.614186:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12055 00000100:00080000:0.0:1787758998.614189:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758998.614190:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758998.614193:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758998.614200:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655236608/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758998.614207:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758998.614209:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758998.614211:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758998.614213:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12055 00000100:00080000:0.0:1787758998.614215:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758998.614217:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758998.614220:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787758998.614226:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655236736/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758998.614237:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758998.614239:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787758998.614240:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758998.614242:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12055 00000100:00080000:0.0:1787758998.614245:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758998.614247:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758998.614249:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787758998.614261:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588655236864/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758998.614267:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758998.614270:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787758998.614271:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758998.614274:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12055 00000100:00080000:0.0:1787758998.614277:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758998.614280:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758998.614283:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787758998.614289:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655236480:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758998.614292:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588655236992/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758998.614300:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787758998.614302:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787758998.614303:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758998.614306:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12055 00000100:00080000:0.0:1787758998.614309:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758998.614311:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758998.614318:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758998.614324:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655236608:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758998.614330:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588655237120/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787758998.614340:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787758998.614343:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655236736:192.168.201.107@tcp:8:kworker.0 00000100:00080000:0.0:1787758998.614345:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787758998.614349:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787758998.614355:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12055 00000100:00080000:0.0:1787758998.614360:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787758998.614365:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787758998.614370:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787758998.614372:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655236864:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787758998.614386:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655237248/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787758998.614401:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655236992:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787758998.614424:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655237120:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787758998.614435:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655237248:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787758998.742064:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787758999.190087:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787758999.765954:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759000.214048:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759000.790018:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759001.238185:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787759001.686305:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.107:988 00000400:00000100:2.0:1787759001.686366:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.107@tcp at host 192.168.201.107 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787759001.686385:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787759001.686435:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787759001.686445:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787759001.686453:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787759001.686462:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787759001.686470:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787759001.686477:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787759001.686485:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:1.0:1787759001.687009:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758998/real 1787759001] req@ffff8c05a9f76a00 x1874588655236992/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759001.687042:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588655236992/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759001.687056:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759001.687073:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759001.687078:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655236992:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759001.687088:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758998/real 1787759001] req@ffff8c05a9f77b80 x1874588655237120/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759001.687102:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588655237120/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759001.687111:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759001.687115:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759001.687118:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655237120:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759001.687125:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758998/real 1787759001] req@ffff8c05a9f74a80 x1874588655237248/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759001.687137:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655237248/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759001.687144:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759001.687148:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759001.687151:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655237248:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759001.687158:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758998/real 1787759001] req@ffff8c05a9f74380 x1874588655236480/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759001.687168:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655236480/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759001.687175:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759001.687178:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759001.687180:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655236480:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759001.687187:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758998/real 1787759001] req@ffff8c05a9f76680 x1874588655236608/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759001.687198:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655236608/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759001.687206:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759001.687209:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759001.687212:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655236608:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759001.687218:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758998/real 1787759001] req@ffff8c05a9f76300 x1874588655236736/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759001.687229:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655236736/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759001.687236:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759001.687240:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759001.687242:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655236736:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759001.687249:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758998/real 1787759001] req@ffff8c05a9f75c00 x1874588655236864/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759001.687260:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588655236864/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759001.687267:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759001.687270:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759001.687273:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655236864:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759001.687280:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787758998/real 1787759001] req@ffff8c05a9f77800 x1874588655236352/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759001.687292:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655236352/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759001.687299:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759001.687302:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759001.687305:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655236352:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787759001.814020:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759002.262063:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759002.837981:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759003.286330:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787759003.734006:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759003.734018:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787759003.734022:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759003.734048:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12060 00000100:00080000:0.0:1787759003.734060:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759003.734068:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787759003.734097:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655237376/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759003.734125:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759003.734129:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787759003.734131:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759003.734135:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12060 00000100:00080000:0.0:1787759003.734138:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759003.734142:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759003.734151:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759003.734167:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588655237504/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759003.734179:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787759003.734180:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655237376:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787759003.734185:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787759003.734189:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759003.734194:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12060 00000100:00080000:0.0:1787759003.734197:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759003.734201:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759003.734206:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759003.734219:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588655237632/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759003.734228:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759003.734231:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787759003.734233:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759003.734237:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12060 00000100:00080000:0.0:1787759003.734241:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759003.734245:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:1.0:1787759003.734250:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:0.0:1787759003.734254:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759003.734267:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588655237760/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759003.734276:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759003.734280:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787759003.734282:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759003.734286:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12060 00000100:00080000:0.0:1787759003.734290:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787759003.734294:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759003/real 1787759003] req@ffff8c05a9f75880 x1874588655237376/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759003.734298:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759003.734305:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759003.734317:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655237376/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759003.734327:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588655237888/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759003.734330:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759003.734340:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787759003.734344:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787759003.734347:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655237376:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787759003.734350:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787759003.734354:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759003.734358:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12060 00000100:00080000:0.0:1787759003.734362:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759003.734366:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759003.734370:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759003.734375:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655237504:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759003.734382:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588655238016/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759003.734393:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759003.734397:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787759003.734400:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787759003.734401:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787759003.734409:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12060 00000100:00080000:0.0:1787759003.734413:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787759003.734418:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759003/real 1787759003] req@ffff8c05a9f76d80 x1874588655237504/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759003.734419:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759003.734426:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759003.734438:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588655237504/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759003.734445:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588655238144/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759003.734449:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759003.734457:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787759003.734460:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787759003.734463:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655237504:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787759003.734465:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787759003.734469:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787759003.734474:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655237632:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787759003.734478:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12060 00000100:00080000:0.0:1787759003.734483:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759003.734488:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759003.734492:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787759003.734497:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759003.734506:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655238272/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759003.734512:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759003/real 1787759003] req@ffff8c05a9f75500 x1874588655237632/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759003.734529:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588655237632/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759003.734537:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759003.734541:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759003.734545:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655237632:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759003.734551:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655237760:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759003.734567:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759003.734576:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759003/real 1787759003] req@ffff8c05a9f74000 x1874588655237760/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759003.734586:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588655237760/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759003.734594:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759003.734597:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759003.734600:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655237760:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759003.734605:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655237888:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759003.734618:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759003.734627:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759003/real 1787759003] req@ffff8c05a9f75180 x1874588655237888/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759003.734637:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588655237888/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759003.734645:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759003.734647:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759003.734650:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655237888:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759003.734665:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655238016:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759003.734681:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759003.734690:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759003/real 1787759003] req@ffff8c05a9f74e00 x1874588655238016/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759003.734700:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588655238016/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759003.734708:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759003.734711:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759003.734714:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655238016:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759003.734719:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655238144:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759003.734732:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759003.734740:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759003/real 1787759003] req@ffff8c05a9f77100 x1874588655238144/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759003.734750:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588655238144/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759003.734763:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759003.734766:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759003.734769:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655238144:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759003.734774:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655238272:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759003.734789:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759003.734797:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759003/real 1787759003] req@ffff8c05a9f75880 x1874588655238272/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759003.734808:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655238272/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759003.734822:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759003.734825:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759003.734828:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655238272:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759003.861965:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759004.310099:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759004.886081:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759005.334164:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759005.910023:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759006.358084:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759006.934013:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759007.382122:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759007.958033:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759008.406061:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787759008.854026:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759008.854034:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787759008.854039:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759008.854048:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12065 00000100:00080000:2.0:1787759008.854058:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759008.854064:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759008.854099:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73800 x1874588655238400/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759008.854121:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759008.854124:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759008.854125:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759008.854128:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12065 00000100:00080000:2.0:1787759008.854130:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759008.854132:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759008.854140:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759008.854151:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588655238528/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759008.854158:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759008.854160:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759008.854163:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787759008.854165:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655238400:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787759008.854167:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12065 00000100:00080000:2.0:1787759008.854173:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759008.854177:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759008.854180:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759008.854188:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588655238656/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759008.854198:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759008.854201:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759008.854202:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759008.854209:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12065 00000100:00080000:2.0:1787759008.854211:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759008.854213:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759008.854216:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759008.854224:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588655238784/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787759008.854232:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787759008.854234:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759008.854238:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759008.854239:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759008.854243:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12065 00000100:00080000:2.0:1787759008.854245:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759008.854248:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759008.854251:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759008.854256:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588655238912/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759008.854261:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759008.854264:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759008.854266:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759008.854268:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12065 00000100:00080000:2.0:1787759008.854272:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787759008.854275:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759008/real 1787759008] req@ffff8c05b2e73800 x1874588655238400/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759008.854278:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759008.854284:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759008.854294:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588655239040/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759008.854298:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73800 x1874588655238400/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759008.854309:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759008.854313:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787759008.854315:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759008.854316:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759008.854319:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12065 00000100:00080000:1.0:1787759008.854323:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787759008.854324:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787759008.854330:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655238400:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787759008.854331:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759008.854335:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759008.854342:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588655239168/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759008.854348:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759008.854350:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759008.854351:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759008.854353:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12065 00000100:00080000:2.0:1787759008.854355:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759008.854357:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759008.854360:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759008.854363:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655238528:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759008.854369:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588655239296/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787759008.854390:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759008.854402:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759008/real 1787759008] req@ffff8c05b2e70e00 x1874588655238528/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759008.854410:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588655238528/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759008.854420:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759008.854422:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759008.854424:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655238528:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759008.854428:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655238656:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759008.854440:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759008.854446:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759008/real 1787759008] req@ffff8c05b2e71880 x1874588655238656/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759008.854453:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588655238656/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759008.854458:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759008.854460:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759008.854462:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655238656:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759008.854465:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655238784:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759008.854475:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759008.854481:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759008/real 1787759008] req@ffff8c05b2e71c00 x1874588655238784/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759008.854487:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588655238784/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759008.854492:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759008.854494:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759008.854495:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655238784:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759008.854498:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655238912:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759008.854507:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759008.854512:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759008/real 1787759008] req@ffff8c05b2e73b80 x1874588655238912/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759008.854519:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588655238912/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759008.854523:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759008.854525:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759008.854526:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655238912:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759008.854529:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655239040:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759008.854538:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759008.854543:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759008/real 1787759008] req@ffff8c05b2e71500 x1874588655239040/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759008.854553:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588655239040/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759008.854558:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759008.854559:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759008.854561:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655239040:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759008.854564:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655239168:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759008.854573:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759008.854578:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759008/real 1787759008] req@ffff8c05b2e73480 x1874588655239168/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759008.854584:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588655239168/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759008.854588:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759008.854590:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759008.854591:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655239168:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759008.854604:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655239296:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759008.854620:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759008.854630:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759008/real 1787759008] req@ffff8c05b2e70700 x1874588655239296/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759008.854641:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588655239296/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759008.854653:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759008.854656:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759008.854658:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655239296:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759008.981956:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759009.430116:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759010.005971:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759010.454055:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759011.030000:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759011.478168:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759012.054029:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759012.502056:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759013.077979:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759013.526102:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787759013.973979:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759013.973988:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787759013.973992:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759013.974001:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12070 00000100:00080000:2.0:1787759013.974011:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759013.974017:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759013.974055:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588655239424/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759013.974079:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759013.974082:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759013.974083:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759013.974086:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12070 00000100:00080000:2.0:1787759013.974088:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759013.974090:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759013.974098:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759013.974106:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588655239552/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759013.974116:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759013.974120:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759013.974122:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759013.974126:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12070 00000100:00080000:2.0:1787759013.974130:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759013.974135:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759013.974138:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655239424:192.168.201.107@tcp:250:kworker.0 00000008:00100000:2.0:1787759013.974146:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759013.974157:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588655239680/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759013.974165:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759013.974167:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759013.974168:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759013.974170:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12070 00000100:00080000:2.0:1787759013.974173:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759013.974175:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759013.974178:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759013.974184:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588655239808/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759013.974190:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759013.974192:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759013.974193:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759013.974196:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12070 00000100:00080000:2.0:1787759013.974198:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759013.974200:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759013.974203:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787759013.974210:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759013.974210:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588655239936/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759013.974216:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759013.974218:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759013.974221:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759013.974225:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12070 00000100:00080000:2.0:1787759013.974227:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759013.974229:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759013.974232:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759013.974239:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588655240064/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759013.974244:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759013.974245:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787759013.974246:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759013.974248:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12070 00000100:00080000:2.0:1787759013.974251:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759013.974258:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759013.974261:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759013.974265:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759013/real 1787759013] req@ffff8c05b2e70380 x1874588655239424/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759013.974271:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588655240192/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759013.974292:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759013.974294:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759013.974296:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759013.974298:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12070 00000100:00080000:2.0:1787759013.974301:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787759013.974303:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588655239424/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759013.974305:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759013.974310:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759013.974318:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588655240320/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759013.974322:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759013.974328:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759013.974333:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655239424:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787759013.974341:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655239552:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759013.974369:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759013.974381:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759013/real 1787759013] req@ffff8c05b2e72d80 x1874588655239552/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759013.974393:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588655239552/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759013.974401:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759013.974404:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759013.974407:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655239552:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759013.974428:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655239680:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759013.974444:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759013.974453:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759013/real 1787759013] req@ffff8c05b2e71180 x1874588655239680/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759013.974474:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588655239680/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759013.974481:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759013.974485:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759013.974488:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655239680:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759013.974493:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655239808:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759013.974508:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759013.974524:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759013/real 1787759013] req@ffff8c05b2e72680 x1874588655239808/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759013.974540:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588655239808/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759013.974548:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759013.974550:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759013.974554:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655239808:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759013.974563:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655239936:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759013.974580:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759013.974589:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759013/real 1787759013] req@ffff8c05b2e72a00 x1874588655239936/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759013.974600:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588655239936/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759013.974607:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759013.974610:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759013.974613:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655239936:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759013.974618:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655240064:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759013.974632:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759013.974641:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759013/real 1787759013] req@ffff8c05b2e72300 x1874588655240064/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759013.974652:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588655240064/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759013.974665:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759013.974668:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759013.974671:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655240064:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759013.974677:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655240192:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759013.974693:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759013.974702:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759013/real 1787759013] req@ffff8c05b2e70a80 x1874588655240192/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759013.974719:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588655240192/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759013.974727:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759013.974729:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759013.974732:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655240192:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759013.974738:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655240320:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759013.974752:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759013.974760:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759013/real 1787759013] req@ffff8c05b2e70000 x1874588655240320/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759013.974770:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588655240320/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759013.974778:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759013.974781:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759013.974784:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655240320:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759014.101987:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759014.550004:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759015.125978:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759015.574130:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759016.149977:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759016.598048:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759017.174053:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759017.622104:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759018.197937:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759018.646050:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787759019.093984:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759019.093991:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787759019.093995:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759019.094004:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12075 00000100:00080000:0.0:1787759019.094014:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759019.094020:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787759019.094062:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655240448/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759019.094083:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759019.094085:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787759019.094087:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759019.094090:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12075 00000100:00080000:0.0:1787759019.094092:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759019.094094:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759019.094102:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759019.094111:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588655240576/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759019.094118:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759019.094122:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787759019.094125:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787759019.094125:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655240448:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787759019.094131:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12075 00000100:00080000:0.0:1787759019.094134:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759019.094136:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759019.094139:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759019.094146:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655240704/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759019.094152:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759019.094154:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787759019.094156:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759019.094159:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12075 00000100:00080000:0.0:1787759019.094161:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759019.094163:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759019.094166:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759019.094172:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655240832/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759019.094178:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759019.094180:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787759019.094182:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759019.094184:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12075 00000100:00080000:0.0:1787759019.094186:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759019.094189:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759019.094191:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787759019.094198:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759019.094199:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655240960/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759019.094206:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759019.094209:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787759019.094210:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759019.094212:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12075 00000100:00080000:0.0:1787759019.094214:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759019.094216:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759019.094219:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759019.094225:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655241088/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759019.094231:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759019.094238:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787759019.094240:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759019.094242:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12075 00000100:00080000:0.0:1787759019.094245:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759019.094248:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759019.094252:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759019.094256:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759019/real 1787759019] req@ffff8c05a9f77800 x1874588655240448/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759019.094260:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588655241216/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759019.094274:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759019.094276:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787759019.094279:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759019.094281:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12075 00000100:00080000:0.0:1787759019.094283:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759019.094284:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759019.094287:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759019.094292:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655240448/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759019.094295:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588655241344/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759019.094313:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759019.094317:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759019.094321:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655240448:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787759019.094341:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655240576:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759019.094357:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759019.094365:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759019/real 1787759019] req@ffff8c05a9f75c00 x1874588655240576/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759019.094373:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588655240576/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759019.094378:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759019.094381:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759019.094383:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655240576:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759019.094386:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655240704:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759019.094395:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759019.094400:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759019/real 1787759019] req@ffff8c05a9f76300 x1874588655240704/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759019.094406:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655240704/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759019.094411:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759019.094413:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759019.094414:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655240704:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759019.094418:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655240832:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759019.094428:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759019.094433:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759019/real 1787759019] req@ffff8c05a9f76680 x1874588655240832/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759019.094440:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655240832/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759019.094445:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759019.094446:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759019.094448:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655240832:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759019.094451:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655240960:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759019.094460:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759019.094465:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759019/real 1787759019] req@ffff8c05a9f74380 x1874588655240960/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759019.094471:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655240960/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759019.094476:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759019.094477:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759019.094479:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655240960:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759019.094482:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655241088:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759019.094490:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759019.094495:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759019/real 1787759019] req@ffff8c05a9f74a80 x1874588655241088/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759019.094502:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655241088/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759019.094506:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759019.094508:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759019.094509:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655241088:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759019.094512:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655241216:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759019.094520:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759019.094525:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759019/real 1787759019] req@ffff8c05a9f77b80 x1874588655241216/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759019.094530:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588655241216/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759019.094535:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759019.094536:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759019.094538:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655241216:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759019.094540:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655241344:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759019.094548:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759019.094553:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759019/real 1787759019] req@ffff8c05a9f76a00 x1874588655241344/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759019.094559:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588655241344/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759019.094569:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759019.094571:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759019.094574:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655241344:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759019.221997:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759019.670095:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759020.246008:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759020.694025:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759021.269986:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759021.718101:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759022.293989:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759022.742068:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759023.318001:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759023.766130:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787759024.214038:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759024.214048:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787759024.214052:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759024.214063:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12080 00000100:00080000:3.0:1787759024.214075:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759024.214085:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787759024.214128:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655241472/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759024.214152:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759024.214154:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759024.214156:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759024.214159:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12080 00000100:00080000:3.0:1787759024.214163:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759024.214167:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759024.214175:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759024.214184:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655241472:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787759024.214187:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655241600/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759024.214203:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759024.214208:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759024.214210:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759024.214214:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12080 00000100:00080000:3.0:1787759024.214218:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759024.214222:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759024.214228:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759024.214240:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655241728/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759024.214248:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759024.214251:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759024.214252:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759024.214255:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12080 00000100:00080000:3.0:1787759024.214258:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759024.214260:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759024.214264:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787759024.214267:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759024.214275:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655241856/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759024.214283:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759024.214286:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759024.214287:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759024.214291:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12080 00000100:00080000:3.0:1787759024.214294:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759024.214298:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759024.214303:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759024.214313:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759024/real 1787759024] req@ffff8c05a9f7a300 x1874588655241472/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759024.214318:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655241984/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759024.214336:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787759024.214337:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655241472/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759024.214346:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759024.214350:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759024.214357:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12080 00000100:00080000:3.0:1787759024.214360:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759024.214364:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:1.0:1787759024.214365:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787759024.214368:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787759024.214372:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759024.214379:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655241472:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787759024.214386:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655242112/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759024.214398:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759024.214401:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787759024.214403:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787759024.214411:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655241600:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787759024.214411:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12080 00000100:00080000:3.0:1787759024.214417:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759024.214422:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759024.214427:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759024.214437:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655242240/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787759024.214446:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787759024.214446:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759024.214452:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759024.214454:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759024.214458:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12080 00000100:00080000:3.0:1787759024.214461:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787759024.214464:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759024/real 1787759024] req@ffff8c05a9f79880 x1874588655241600/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759024.214465:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759024.214472:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759024.214484:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655241600/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759024.214486:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655242368/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759024.214524:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759024.214529:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759024.214533:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655241600:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759024.214539:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655241728:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759024.214554:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759024.214561:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759024/real 1787759024] req@ffff8c05a9f78700 x1874588655241728/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759024.214569:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655241728/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759024.214574:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759024.214576:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759024.214577:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655241728:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759024.214581:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655241856:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759024.214592:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759024.214598:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759024/real 1787759024] req@ffff8c05a9f7b480 x1874588655241856/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759024.214605:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655241856/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759024.214610:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759024.214612:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759024.214615:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655241856:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759024.214618:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655241984:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759024.214640:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759024.214647:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759024/real 1787759024] req@ffff8c05a9f78380 x1874588655241984/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759024.214654:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655241984/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759024.214660:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759024.214662:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759024.214678:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655241984:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759024.214683:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655242112:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759024.214698:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759024.214706:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759024/real 1787759024] req@ffff8c05a9f78a80 x1874588655242112/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759024.214713:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655242112/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759024.214719:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759024.214721:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759024.214723:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655242112:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759024.214737:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655242240:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759024.214748:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759024.214755:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759024/real 1787759024] req@ffff8c05a9f7a680 x1874588655242240/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759024.214762:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655242240/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759024.214768:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759024.214770:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759024.214771:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655242240:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759024.214775:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655242368:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759024.214784:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759024.214790:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759024/real 1787759024] req@ffff8c05a9f7aa00 x1874588655242368/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759024.214797:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655242368/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759024.214802:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759024.214804:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759024.214806:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655242368:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759024.341966:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759024.790083:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759025.366005:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759025.814145:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759026.389969:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759026.838042:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759027.413982:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759027.862169:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759028.437964:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759028.886045:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787759029.334647:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759029.334660:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787759029.334665:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759029.334678:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12085 00000100:00080000:1.0:1787759029.334691:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759029.334702:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759029.334755:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588655242496/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759029.334790:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759029.334795:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759029.334797:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759029.334802:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12085 00000100:00080000:1.0:1787759029.334807:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759029.334812:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759029.334822:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759029.334838:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655242624/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759029.334852:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759029.334856:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759029.334858:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759029.334866:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12085 00000100:00080000:1.0:1787759029.334870:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759029.334929:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759029.334941:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759029.334956:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655242752/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759029.334966:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759029.334969:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759029.334970:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787759029.334973:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655242496:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787759029.334974:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12085 00000100:00080000:1.0:1787759029.334988:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759029.334996:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759029.335002:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759029.335027:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655242880/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759029.335039:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759029.335042:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759029.335043:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759029.335046:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12085 00000100:00080000:1.0:1787759029.335050:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759029.335055:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759029.335060:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759029.335075:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655243008/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759029.335085:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759029.335088:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759029.335090:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759029.335096:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12085 00000100:00080000:1.0:1787759029.335100:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759029.335104:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759029.335110:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759029.335120:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655243136/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759029.335126:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759029.335128:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787759029.335129:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759029.335132:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12085 00000100:00080000:1.0:1787759029.335134:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759029.335137:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759029.335140:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759029.335148:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655243264/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759029.335155:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759029.335161:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759029.335162:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787759029.335167:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787759029.335172:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12085 00000100:00080000:1.0:1787759029.335178:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759029.335183:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759029.335196:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759029.335210:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655243392/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787759029.335292:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759029/real 1787759029] req@ffff8c05b2d9a300 x1874588655242496/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759029.335310:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588655242496/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759029.335320:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759029.335323:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759029.335326:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655242496:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787759029.335334:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655242624:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787759029.335357:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759029.335367:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759029/real 1787759029] req@ffff8c05b2d98380 x1874588655242624/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759029.335376:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655242624/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759029.335381:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759029.335384:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759029.335386:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655242624:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759029.335407:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655242752:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787759029.335429:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759029.335441:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759029/real 1787759029] req@ffff8c05b2d99f80 x1874588655242752/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759029.335455:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655242752/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759029.335464:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759029.335468:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759029.335471:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655242752:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759029.335476:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655242880:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787759029.335493:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759029.335500:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759029/real 1787759029] req@ffff8c05b2d9aa00 x1874588655242880/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759029.335508:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655242880/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759029.335513:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759029.335515:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759029.335517:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655242880:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759029.335521:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655243008:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787759029.335532:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759029.335539:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759029/real 1787759029] req@ffff8c05b2d9b480 x1874588655243008/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759029.335546:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655243008/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759029.335551:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759029.335553:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759029.335555:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655243008:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759029.335558:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655243136:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787759029.335569:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759029.336681:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759029/real 1787759029] req@ffff8c05b2d98700 x1874588655243136/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759029.336697:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655243136/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759029.336703:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759029.336706:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759029.336708:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655243136:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759029.336713:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655243264:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787759029.336739:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759029.336752:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759029/real 1787759029] req@ffff8c05b2d98000 x1874588655243264/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759029.336771:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655243264/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759029.336780:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759029.336783:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759029.336787:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655243264:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759029.336793:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655243392:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787759029.336816:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759029.336828:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759029/real 1787759029] req@ffff8c05b2d9b100 x1874588655243392/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759029.336836:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655243392/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759029.336842:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759029.336844:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759029.336846:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655243392:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759029.462023:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759029.910151:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759030.485947:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759030.934053:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759031.510024:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759031.958099:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759032.533980:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759032.982076:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759033.558013:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759034.006146:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787759034.453973:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759034.453991:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787759034.453996:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759034.454007:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12090 00000100:00080000:1.0:1787759034.454019:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759034.454038:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759034.454083:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655243520/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759034.454108:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759034.454111:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759034.454113:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759034.454117:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12090 00000100:00080000:1.0:1787759034.454121:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759034.454125:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759034.454135:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759034.454148:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655243648/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787759034.454156:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655243520:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787759034.454157:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759034.454161:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759034.454162:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759034.454165:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12090 00000100:00080000:1.0:1787759034.454168:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759034.454171:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759034.454175:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759034.454186:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655243776/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759034.454195:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759034.454197:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759034.454199:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759034.454201:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12090 00000100:00080000:1.0:1787759034.454204:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759034.454207:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759034.454211:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759034.454219:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655243904/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787759034.454222:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787759034.454227:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759034.454229:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759034.454231:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759034.454233:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12090 00000100:00080000:1.0:1787759034.454236:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759034.454239:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759034.454243:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759034.454259:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655244032/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787759034.454264:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759034/real 1787759034] req@ffff8c05b2d9bb80 x1874588655243520/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759034.454267:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759034.454271:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759034.454272:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759034.454276:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12090 00000100:00080000:1.0:1787759034.454279:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787759034.454281:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655243520/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759034.454282:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759034.454286:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787759034.454291:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787759034.454294:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655244160/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759034.454295:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759034.454299:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655243520:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787759034.454302:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759034.454305:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787759034.454306:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759034.454310:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12090 00000100:00080000:1.0:1787759034.454312:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759034.454316:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759034.454319:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759034.454329:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655244288/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787759034.454332:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655243648:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787759034.454337:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759034.454340:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759034.454341:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759034.454344:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12090 00000100:00080000:1.0:1787759034.454346:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759034.454350:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:3.0:1787759034.454352:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:1.0:1787759034.454354:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759034.454364:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655244416/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787759034.454364:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759034/real 1787759034] req@ffff8c05b2d99880 x1874588655243648/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759034.454375:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655243648/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759034.454383:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759034.454385:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759034.454388:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655243648:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759034.454393:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655243776:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787759034.454412:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759034.454421:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759034/real 1787759034] req@ffff8c05b2d9b100 x1874588655243776/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759034.454429:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655243776/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759034.454438:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759034.454440:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759034.454442:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655243776:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759034.454445:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655243904:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787759034.454473:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759034.454483:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759034/real 1787759034] req@ffff8c05b2d98000 x1874588655243904/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759034.454505:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655243904/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759034.454512:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759034.454515:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759034.454517:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655243904:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759034.454521:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655244032:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787759034.454536:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759034.454542:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759034/real 1787759034] req@ffff8c05b2d98700 x1874588655244032/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759034.454549:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655244032/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759034.454554:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759034.454556:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759034.454558:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655244032:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759034.454562:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655244160:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787759034.454571:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759034.454576:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759034/real 1787759034] req@ffff8c05b2d9b480 x1874588655244160/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759034.454582:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655244160/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759034.454587:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759034.454588:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759034.454589:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655244160:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759034.454600:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655244288:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787759034.454609:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759034.454614:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759034/real 1787759034] req@ffff8c05b2d9aa00 x1874588655244288/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759034.454621:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655244288/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759034.454625:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759034.454627:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759034.454629:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655244288:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759034.454631:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655244416:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787759034.454640:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759034.454645:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759034/real 1787759034] req@ffff8c05b2d99f80 x1874588655244416/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759034.454652:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655244416/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759034.454656:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759034.454657:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759034.454659:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655244416:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759034.581983:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759035.030045:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759035.605983:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759036.054103:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759036.629951:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759037.078062:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759037.654047:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759038.102094:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759038.678194:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759039.126064:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787759039.573985:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759039.573996:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787759039.574000:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759039.574017:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12095 00000100:00080000:2.0:1787759039.574028:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759039.574034:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759039.574069:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588655244544/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759039.574100:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759039.574105:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759039.574107:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759039.574112:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12095 00000100:00080000:2.0:1787759039.574116:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759039.574120:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759039.574130:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759039.574152:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588655244672/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787759039.574153:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655244544:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787759039.574169:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759039.574172:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759039.574174:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759039.574177:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12095 00000100:00080000:2.0:1787759039.574180:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759039.574184:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759039.574188:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759039.574198:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588655244800/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759039.574205:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759039.574208:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759039.574210:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759039.574212:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12095 00000100:00080000:2.0:1787759039.574214:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759039.574217:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759039.574222:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787759039.574222:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759039.574238:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588655244928/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759039.574249:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759039.574253:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759039.574255:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759039.574260:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12095 00000100:00080000:2.0:1787759039.574264:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759039.574269:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787759039.574274:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759039/real 1787759039] req@ffff8c05b2e73100 x1874588655244544/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787759039.574275:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759039.574296:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588655245056/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787759039.574299:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588655244544/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759039.574316:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759039.574320:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759039.574321:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759039.574322:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759039.574327:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787759039.574329:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12095 00000100:00080000:2.0:1787759039.574334:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787759039.574335:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655244544:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787759039.574342:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759039.574346:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759039.574354:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588655245184/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787759039.574362:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655244672:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787759039.574363:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759039.574370:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787759039.574374:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759039.574378:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12095 00000100:00080000:2.0:1787759039.574381:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759039.574383:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759039.574387:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787759039.574393:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759039.574412:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588655245312/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759039.574420:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759039.574423:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759039.574424:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759039.574431:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12095 00000100:00100000:3.0:1787759039.574432:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759039/real 1787759039] req@ffff8c05b2e70000 x1874588655244672/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759039.574436:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759039.574444:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787759039.574460:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588655244672/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787759039.574466:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787759039.574477:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759039.574482:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759039.574486:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588655245440/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787759039.574490:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655244672:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759039.574506:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655244800:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787759039.574528:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759039.574537:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759039/real 1787759039] req@ffff8c05b2e70a80 x1874588655244800/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759039.574545:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588655244800/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759039.574551:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759039.574553:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759039.574555:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655244800:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759039.574559:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655244928:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787759039.574570:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759039.574576:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759039/real 1787759039] req@ffff8c05b2e72300 x1874588655244928/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759039.574588:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588655244928/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759039.574593:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759039.574595:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759039.574597:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655244928:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759039.574600:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655245056:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787759039.574611:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759039.574617:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759039/real 1787759039] req@ffff8c05b2e72a00 x1874588655245056/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759039.574624:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588655245056/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759039.574629:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759039.574631:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759039.574632:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655245056:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759039.574644:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655245184:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787759039.574655:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759039.574661:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759039/real 1787759039] req@ffff8c05b2e72680 x1874588655245184/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759039.574668:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588655245184/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759039.574678:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759039.574680:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759039.574683:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655245184:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759039.574686:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655245312:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787759039.574696:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759039.574701:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759039/real 1787759039] req@ffff8c05b2e71180 x1874588655245312/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759039.574709:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588655245312/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759039.574714:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759039.574715:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759039.574717:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655245312:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759039.574720:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655245440:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787759039.574737:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759039.574758:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759039/real 1787759039] req@ffff8c05b2e72d80 x1874588655245440/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759039.574770:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588655245440/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759039.574779:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759039.574782:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759039.574786:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655245440:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759039.702013:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759040.150142:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759040.725977:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759041.174037:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759041.749993:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759042.198111:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759042.773968:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759043.222014:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759043.798001:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759044.246170:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787759044.694003:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759044.694013:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787759044.694017:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759044.694026:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12101 00000100:00080000:2.0:1787759044.694038:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759044.694044:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759044.694091:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588655245568/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759044.694118:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759044.694121:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759044.694122:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759044.694125:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12101 00000100:00080000:2.0:1787759044.694127:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759044.694130:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759044.694137:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759044.694147:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588655245696/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759044.694154:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759044.694157:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759044.694158:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759044.694161:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12101 00000100:00080000:2.0:1787759044.694164:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759044.694167:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759044.694170:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759044.694180:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588655245824/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787759044.694181:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655245568:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787759044.694199:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759044.694203:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759044.694205:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759044.694210:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12101 00000100:00080000:2.0:1787759044.694214:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759044.694218:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759044.694224:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759044.694238:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588655245952/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759044.694245:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759044.694248:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759044.694250:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759044.694252:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12101 00000100:00080000:2.0:1787759044.694256:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759044.694259:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759044.694263:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787759044.694264:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759044.694277:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588655246080/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759044.694294:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759044.694316:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759044.694318:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759044.694324:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12101 00000100:00080000:2.0:1787759044.694329:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759044.694333:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759044.694339:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759044.694347:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759044/real 1787759044] req@ffff8c05b2e70380 x1874588655245568/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759044.694355:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588655246208/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759044.694377:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759044.694381:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787759044.694384:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787759044.694384:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588655245568/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759044.694392:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12101 00000100:00080000:2.0:1787759044.694398:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759044.694404:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:3.0:1787759044.694408:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787759044.694409:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787759044.694419:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759044.694423:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588655246336/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787759044.694430:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655245568:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787759044.694434:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759044.694440:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759044.694441:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759044.694444:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12101 00000100:00100000:3.0:1787759044.694449:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655245696:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787759044.694450:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759044.694454:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759044.694461:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759044.694470:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588655246464/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787759044.694490:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759044.694508:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759044/real 1787759044] req@ffff8c05b2e70700 x1874588655245696/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759044.694524:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588655245696/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759044.694534:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759044.694539:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759044.694543:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655245696:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759044.694570:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655245824:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787759044.694594:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759044.694607:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759044/real 1787759044] req@ffff8c05b2e73480 x1874588655245824/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759044.694621:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588655245824/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759044.694631:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759044.694634:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759044.694638:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655245824:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759044.694645:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655245952:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787759044.694668:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759044.694680:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759044/real 1787759044] req@ffff8c05b2e71500 x1874588655245952/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759044.694694:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588655245952/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759044.694703:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759044.694707:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759044.694710:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655245952:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759044.694717:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655246080:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787759044.694739:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759044.694751:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759044/real 1787759044] req@ffff8c05b2e73b80 x1874588655246080/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759044.694764:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588655246080/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759044.694773:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759044.694776:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759044.694780:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655246080:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759044.694786:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655246208:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787759044.694808:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759044.694821:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759044/real 1787759044] req@ffff8c05b2e71c00 x1874588655246208/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759044.694833:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588655246208/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759044.694841:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759044.694845:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759044.694849:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655246208:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759044.694855:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655246336:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787759044.694902:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759044.694916:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759044/real 1787759044] req@ffff8c05b2e71880 x1874588655246336/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759044.694938:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588655246336/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759044.694947:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759044.694950:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759044.694954:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655246336:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759044.694960:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655246464:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787759044.694982:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759044.694994:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759044/real 1787759044] req@ffff8c05b2e70e00 x1874588655246464/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759044.695007:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588655246464/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759044.695016:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759044.695020:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759044.695024:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655246464:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759044.821969:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759045.270138:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759045.846017:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759046.294146:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759046.870009:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759047.318111:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759047.893934:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759048.342111:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759048.917982:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759049.366097:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787759049.814020:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759049.814032:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787759049.814036:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759049.814048:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12106 00000100:00080000:2.0:1787759049.814059:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759049.814066:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759049.814110:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73800 x1874588655246592/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759049.814144:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759049.814148:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759049.814150:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759049.814155:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12106 00000100:00080000:2.0:1787759049.814159:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759049.814163:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759049.814172:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759049.814190:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588655246720/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787759049.814199:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655246592:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787759049.814206:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759049.814211:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759049.814229:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759049.814234:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12106 00000100:00080000:2.0:1787759049.814239:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759049.814243:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759049.814248:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759049.814262:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588655246848/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759049.814274:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759049.814278:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759049.814280:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759049.814283:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12106 00000100:00080000:2.0:1787759049.814287:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759049.814293:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:3.0:1787759049.814295:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:2.0:1787759049.814300:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759049.814312:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588655246976/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759049.814323:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759049.814327:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759049.814329:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759049.814333:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12106 00000100:00080000:2.0:1787759049.814337:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759049.814342:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759049.814348:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759049.814356:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759049/real 1787759049] req@ffff8c05b2e73800 x1874588655246592/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759049.814362:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588655247104/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759049.814380:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759049.814385:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1787759049.814385:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73800 x1874588655246592/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759049.814390:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759049.814401:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12106 00000100:00080000:3.0:1787759049.814404:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759049.814406:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759049.814413:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787759049.814414:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759049.814423:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759049.814424:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655246592:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787759049.814438:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588655247232/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759049.814449:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759049.814453:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787759049.814455:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759049.814458:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12106 00000100:00080000:2.0:1787759049.814462:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787759049.814467:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655246720:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787759049.814468:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759049.814478:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759049.814495:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588655247360/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787759049.814504:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787759049.814516:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787759049.814519:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759049/real 1787759049] req@ffff8c05b2e70e00 x1874588655246720/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759049.814522:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759049.814525:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759049.814531:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12106 00000100:00080000:2.0:1787759049.814539:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759049.814545:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787759049.814548:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588655246720/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787759049.814554:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787759049.814558:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759049.814562:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759049.814566:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655246720:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759049.814570:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588655247488/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787759049.814575:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655246848:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787759049.814599:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759049.814608:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759049/real 1787759049] req@ffff8c05b2e71880 x1874588655246848/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759049.814616:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588655246848/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759049.814627:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759049.814629:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759049.814636:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655246848:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759049.814640:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655246976:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787759049.814653:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759049.814659:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759049/real 1787759049] req@ffff8c05b2e71c00 x1874588655246976/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759049.814666:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588655246976/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759049.814672:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759049.814674:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759049.814676:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655246976:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759049.814679:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655247104:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787759049.814689:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759049.814694:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759049/real 1787759049] req@ffff8c05b2e73b80 x1874588655247104/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759049.814701:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588655247104/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759049.814706:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759049.814708:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759049.814711:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655247104:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759049.814714:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655247232:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787759049.814723:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759049.814729:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759049/real 1787759049] req@ffff8c05b2e71500 x1874588655247232/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759049.814735:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588655247232/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759049.814740:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759049.814741:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759049.814743:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655247232:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759049.814756:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655247360:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787759049.814766:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759049.814772:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759049/real 1787759049] req@ffff8c05b2e73480 x1874588655247360/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759049.814779:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588655247360/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759049.814784:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759049.814785:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759049.814787:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655247360:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759049.814791:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655247488:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787759049.814800:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759049.814806:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759049/real 1787759049] req@ffff8c05b2e70700 x1874588655247488/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759049.814813:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588655247488/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759049.814818:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759049.814820:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759049.814822:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655247488:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759049.941971:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759050.390217:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759050.966037:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759051.414083:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759051.990013:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759052.438166:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759053.014139:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759053.462139:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759054.038022:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759054.486171:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787759054.934037:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759054.934047:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787759054.934051:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759054.934061:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12111 00000100:00080000:2.0:1787759054.934072:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759054.934079:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759054.934119:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588655247616/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759054.934159:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759054.934162:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759054.934164:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759054.934168:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12111 00000100:00080000:2.0:1787759054.934171:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759054.934174:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759054.934183:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759054.934195:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588655247744/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759054.934208:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759054.934211:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759054.934214:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787759054.934220:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655247616:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787759054.934221:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12111 00000100:00080000:2.0:1787759054.934229:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759054.934235:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759054.934256:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759054.934272:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588655247872/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759054.934283:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759054.934287:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759054.934289:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759054.934293:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12111 00000100:00080000:2.0:1787759054.934298:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759054.934303:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759054.934308:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787759054.934317:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759054.934323:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588655248000/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759054.934335:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759054.934339:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759054.934341:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759054.934346:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12111 00000100:00080000:2.0:1787759054.934351:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759054.934356:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759054.934362:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759054.934375:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588655248128/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787759054.934384:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759054/real 1787759054] req@ffff8c05b2e70380 x1874588655247616/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759054.934387:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759054.934399:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759054.934404:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759054.934411:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12111 00000100:00100000:3.0:1787759054.934415:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588655247616/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759054.934417:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759054.934428:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:3.0:1787759054.934431:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787759054.934436:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787759054.934436:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759054.934444:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655247616:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787759054.934456:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588655248256/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759054.934466:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759054.934470:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787759054.934473:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787759054.934476:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655247744:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787759054.934480:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12111 00000100:00080000:2.0:1787759054.934488:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759054.934494:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759054.934500:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759054.934516:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588655248384/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787759054.934516:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759054.934528:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759054/real 1787759054] req@ffff8c05b2e72d80 x1874588655247744/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759054.934555:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588655247744/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759054.934556:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759054.934568:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759054.934574:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759054.934575:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759054.934583:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787759054.934585:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12111 00000100:00100000:3.0:1787759054.934589:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655247744:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787759054.934595:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787759054.934598:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655247872:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787759054.934612:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:3.0:1787759054.934616:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:2.0:1787759054.934623:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759054.934627:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759054/real 1787759054] req@ffff8c05b2e71180 x1874588655247872/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759054.934640:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588655247872/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759054.934644:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588655248512/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759054.934650:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759054.934659:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759054.934665:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655247872:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759054.934672:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655248000:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787759054.934684:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759054.934690:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759054/real 1787759054] req@ffff8c05b2e72680 x1874588655248000/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759054.934703:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588655248000/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759054.934709:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759054.934711:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759054.934713:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655248000:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759054.934716:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655248128:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787759054.934726:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759054.934732:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759054/real 1787759054] req@ffff8c05b2e72a00 x1874588655248128/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759054.934744:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588655248128/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759054.934752:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759054.934755:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759054.934759:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655248128:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759054.934764:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655248256:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787759054.934785:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759054.934796:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759054/real 1787759054] req@ffff8c05b2e72300 x1874588655248256/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759054.934804:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588655248256/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759054.934814:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759054.934816:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759054.934819:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655248256:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759054.934833:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655248384:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787759054.934847:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759054.934853:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759054/real 1787759054] req@ffff8c05b2e70a80 x1874588655248384/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759054.934861:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588655248384/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759054.934866:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759054.934868:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759054.934870:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655248384:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759054.934897:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655248512:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787759054.934918:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759054.934930:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759054/real 1787759054] req@ffff8c05b2e70000 x1874588655248512/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759054.934943:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588655248512/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759054.934952:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759054.934955:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759054.934959:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655248512:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759055.062003:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759055.510057:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759056.086021:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759056.534151:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759057.110026:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759057.558136:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759058.134002:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759058.582096:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759059.158009:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759059.606030:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787759060.053983:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759060.053993:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787759060.053997:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759060.054006:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12116 00000100:00080000:1.0:1787759060.054017:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759060.054022:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759060.054066:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655248640/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759060.054087:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759060.054089:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759060.054090:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759060.054093:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12116 00000100:00080000:1.0:1787759060.054095:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759060.054098:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759060.054106:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759060.054116:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588655248768/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759060.054122:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759060.054124:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759060.054125:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759060.054128:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12116 00000100:00080000:1.0:1787759060.054129:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759060.054131:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759060.054134:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759060.054142:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655248896/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787759060.054142:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655248640:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787759060.054156:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759060.054158:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759060.054159:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759060.054161:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12116 00000100:00080000:1.0:1787759060.054163:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759060.054164:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759060.054168:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759060.054174:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655249024/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759060.054180:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759060.054182:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759060.054183:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759060.054185:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12116 00000100:00080000:1.0:1787759060.054187:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759060.054190:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759060.054193:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759060.054198:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655249152/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759060.054203:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759060.054205:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759060.054206:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759060.054209:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12116 00000100:00080000:1.0:1787759060.054211:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759060.054213:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759060.054216:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787759060.054220:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759060.054224:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655249280/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759060.054230:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759060.054232:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787759060.054233:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759060.054236:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12116 00000100:00080000:1.0:1787759060.054239:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759060.054241:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759060.054244:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759060.054252:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655249408/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759060.054257:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759060.054259:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759060.054260:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759060.054263:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12116 00000100:00080000:1.0:1787759060.054265:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759060.054269:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787759060.054270:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759060/real 1787759060] req@ffff8c05b2d98380 x1874588655248640/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787759060.054272:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759060.054281:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655249536/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787759060.054298:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655248640/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759060.054310:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759060.054315:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759060.054319:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655248640:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787759060.054326:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655248768:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787759060.054349:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759060.054362:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759060/real 1787759060] req@ffff8c05b2d9a300 x1874588655248768/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759060.054374:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588655248768/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759060.054382:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759060.054386:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759060.054390:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655248768:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759060.054421:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655248896:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787759060.054440:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759060.054451:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759060/real 1787759060] req@ffff8c05b2d99f80 x1874588655248896/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759060.054462:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655248896/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759060.054470:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759060.054480:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759060.054484:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655248896:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759060.054490:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655249024:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787759060.054509:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759060.054521:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759060/real 1787759060] req@ffff8c05b2d9aa00 x1874588655249024/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759060.054532:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655249024/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759060.054541:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759060.054544:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759060.054548:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655249024:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759060.054554:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655249152:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787759060.054571:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759060.054581:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759060/real 1787759060] req@ffff8c05b2d9b480 x1874588655249152/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759060.054592:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655249152/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759060.054600:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759060.054604:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759060.054615:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655249152:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759060.054622:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655249280:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787759060.054640:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759060.054649:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759060/real 1787759060] req@ffff8c05b2d98700 x1874588655249280/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759060.054660:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655249280/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759060.054668:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759060.054672:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759060.054675:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655249280:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759060.054680:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655249408:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787759060.054703:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759060.054714:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759060/real 1787759060] req@ffff8c05b2d98000 x1874588655249408/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759060.054732:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655249408/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759060.054740:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759060.054743:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759060.054746:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655249408:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759060.054752:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655249536:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787759060.054771:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759060.054781:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759060/real 1787759060] req@ffff8c05b2d9b100 x1874588655249536/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759060.054792:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655249536/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759060.054800:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759060.054804:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759060.054807:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655249536:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759060.181929:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759060.630171:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759061.206009:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759061.654123:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759062.230070:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759062.678171:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759063.253987:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759063.702085:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759064.277958:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759064.726115:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787759065.174007:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759065.174018:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787759065.174023:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759065.174033:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12121 00000100:00080000:1.0:1787759065.174044:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759065.174051:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759065.174089:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655249664/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759065.174117:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759065.174120:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759065.174122:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759065.174125:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12121 00000100:00080000:1.0:1787759065.174128:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759065.174132:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759065.174142:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759065.174154:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655249792/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759065.174164:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759065.174168:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759065.174170:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759065.174175:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12121 00000100:00100000:3.0:1787759065.174177:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655249664:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787759065.174178:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759065.174186:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759065.174191:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759065.174203:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655249920/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759065.174212:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759065.174215:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759065.174217:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759065.174220:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12121 00000100:00080000:1.0:1787759065.174223:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759065.174227:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759065.174231:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759065.174241:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655250048/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759065.174249:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759065.174252:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759065.174254:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759065.174257:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12121 00000100:00080000:1.0:1787759065.174261:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759065.174265:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759065.174269:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759065.174279:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655250176/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759065.174287:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759065.174290:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759065.174292:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759065.174296:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12121 00000100:00080000:1.0:1787759065.174299:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759065.174304:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787759065.174304:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655249792:192.168.201.107@tcp:38:kworker.0 00000008:00100000:1.0:1787759065.174311:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759065.174324:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655250304/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759065.174334:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759065.174337:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787759065.174339:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759065.174343:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12121 00000100:00080000:1.0:1787759065.174346:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787759065.174350:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655249920:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787759065.174353:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759065.174362:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759065.174373:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655250432/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759065.174382:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787759065.174385:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655250048:192.168.201.107@tcp:8:kworker.0 00000100:00080000:1.0:1787759065.174387:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759065.174390:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759065.174395:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12121 00000100:00080000:1.0:1787759065.174399:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759065.174403:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759065.174407:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759065.174417:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655250176:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759065.174419:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655250560/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787759065.174452:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655250304:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759065.174476:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655250432:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759065.174501:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655250560:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759065.302001:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759065.750016:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759066.325966:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759066.774112:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759067.349981:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759067.798105:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787759068.246206:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.107:988 00000400:00000100:2.0:1787759068.246259:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.107@tcp at host 192.168.201.107 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787759068.246278:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787759068.246327:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787759068.246338:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787759068.246346:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787759068.246354:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787759068.246362:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787759068.246376:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787759068.246384:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:3.0:1787759068.246820:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759065/real 1787759068] req@ffff8c05b2d9b480 x1874588655250304/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759068.246845:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655250304/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759068.246858:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759068.246864:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759068.246869:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655250304:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759068.246911:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759065/real 1787759068] req@ffff8c05b2d9aa00 x1874588655250432/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759068.246925:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655250432/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759068.246935:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759068.246939:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759068.246942:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655250432:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759068.246949:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759065/real 1787759068] req@ffff8c05b2d99f80 x1874588655250560/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759068.246961:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655250560/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759068.246970:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759068.246974:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759068.246977:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655250560:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759068.246984:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759065/real 1787759068] req@ffff8c05b2d9bb80 x1874588655249792/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759068.246997:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655249792/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759068.247005:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759068.247009:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759068.247011:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655249792:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759068.247018:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759065/real 1787759068] req@ffff8c05b2d9b100 x1874588655249920/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759068.247029:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655249920/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759068.247037:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759068.247041:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759068.247044:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655249920:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759068.247051:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759065/real 1787759068] req@ffff8c05b2d98000 x1874588655250048/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759068.247062:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655250048/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759068.247069:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759068.247072:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759068.247075:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655250048:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759068.247082:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759065/real 1787759068] req@ffff8c05b2d98700 x1874588655250176/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759068.247092:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655250176/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759068.247099:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759068.247101:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759068.247105:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655250176:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759068.247112:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759065/real 1787759068] req@ffff8c05b2d99880 x1874588655249664/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759068.247123:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655249664/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759068.247131:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759068.247134:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759068.247137:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655249664:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787759068.373995:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759068.822120:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759069.397959:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759069.846043:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787759070.293993:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759070.294005:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787759070.294009:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759070.294020:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12126 00000100:00080000:1.0:1787759070.294032:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759070.294040:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759070.294079:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588655250688/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759070.294112:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759070.294115:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759070.294117:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759070.294121:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12126 00000100:00080000:1.0:1787759070.294124:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759070.294128:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759070.294139:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759070.294153:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655250816/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759070.294164:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759070.294168:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759070.294170:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759070.294174:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12126 00000100:00080000:1.0:1787759070.294177:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759070.294181:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759070.294187:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759070.294203:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655250944/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787759070.294206:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655250688:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787759070.294225:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759070.294229:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759070.294231:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759070.294236:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12126 00000100:00080000:1.0:1787759070.294241:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759070.294246:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759070.294251:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759070.294264:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655251072/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759070.294275:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759070.294279:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759070.294281:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759070.294284:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12126 00000100:00080000:1.0:1787759070.294290:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759070.294295:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759070.294300:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787759070.294315:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759070.294317:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655251200/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759070.294331:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759070.294335:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759070.294337:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759070.294341:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12126 00000100:00080000:1.0:1787759070.294346:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759070.294351:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759070.294357:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759070.294372:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655251328/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759070.294393:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787759070.294395:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759070/real 1787759070] req@ffff8c05b2d9a300 x1874588655250688/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759070.294403:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787759070.294411:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787759070.294435:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588655250688/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759070.294439:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12126 00000100:00080000:1.0:1787759070.294453:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759070.294456:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759070.294462:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:3.0:1787759070.294465:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000008:00100000:1.0:1787759070.294475:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759070.294476:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655250688:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787759070.294495:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655250816:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759070.294501:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655251456/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759070.294514:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759070.294518:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759070.294519:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759070.294526:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12126 00000100:00080000:1.0:1787759070.294532:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787759070.294535:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787759070.294546:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759070.294554:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759070.294558:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759070/real 1787759070] req@ffff8c05b2d98380 x1874588655250816/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759070.294572:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655251584/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787759070.294577:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655250816/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759070.294617:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759070.294626:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759070.294633:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655250816:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759070.294675:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655250944:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787759070.294702:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759070.294714:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759070/real 1787759070] req@ffff8c05b2d99880 x1874588655250944/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759070.294726:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655250944/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759070.294734:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759070.294738:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759070.294741:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655250944:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759070.294748:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655251072:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787759070.294769:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759070.296411:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759070/real 1787759070] req@ffff8c05b2d98700 x1874588655251072/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759070.296441:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655251072/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759070.296452:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759070.296456:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759070.296460:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655251072:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759070.296470:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655251200:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787759070.296521:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759070.296542:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759070/real 1787759070] req@ffff8c05b2d98000 x1874588655251200/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759070.296553:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655251200/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759070.296560:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759070.296563:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759070.296566:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655251200:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759070.296572:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655251328:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787759070.296593:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759070.296603:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759070/real 1787759070] req@ffff8c05b2d9b100 x1874588655251328/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759070.296613:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655251328/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759070.296620:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759070.296624:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759070.296627:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655251328:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759070.296631:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655251456:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787759070.296647:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759070.296654:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759070/real 1787759070] req@ffff8c05b2d9bb80 x1874588655251456/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759070.296665:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655251456/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759070.296669:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759070.296671:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759070.296673:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655251456:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759070.296676:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655251584:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787759070.296691:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759070.296699:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759070/real 1787759070] req@ffff8c05b2d99f80 x1874588655251584/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759070.296710:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655251584/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759070.296717:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759070.296720:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759070.296723:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655251584:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759070.422978:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759070.870132:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759071.445985:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759071.894049:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759072.469974:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759072.918147:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759073.493960:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759073.942013:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759074.517963:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759074.966116:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787759075.413996:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759075.414007:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787759075.414013:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759075.414023:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12131 00000100:00080000:1.0:1787759075.414036:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759075.414044:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759075.414087:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655251712/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759075.414116:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759075.414119:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759075.414121:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759075.414125:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12131 00000100:00080000:1.0:1787759075.414129:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759075.414133:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759075.414143:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759075.414155:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655251712:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787759075.414167:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655251840/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759075.414178:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759075.414182:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759075.414184:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759075.414188:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12131 00000100:00080000:1.0:1787759075.414192:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759075.414196:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759075.414201:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759075.414214:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655251968/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787759075.414227:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787759075.414236:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759075.414240:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759075.414242:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759075.414247:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12131 00000100:00080000:1.0:1787759075.414251:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759075.414255:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759075.414260:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759075.414268:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759075/real 1787759075] req@ffff8c05b2d9aa00 x1874588655251712/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759075.414285:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655252096/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787759075.414286:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655251712/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759075.414315:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759075.414317:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759075.414321:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787759075.414324:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1787759075.414328:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655251712:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787759075.414329:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759075.414338:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12131 00000100:00080000:1.0:1787759075.414342:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759075.414346:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759075.414352:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759075.414352:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655251840:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759075.414371:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655252224/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787759075.414377:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787759075.414390:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787759075.414390:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759075/real 1787759075] req@ffff8c05b2d9b480 x1874588655251840/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759075.414399:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759075.414403:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759075.414410:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12131 00000100:00100000:3.0:1787759075.414411:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655251840/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759075.414418:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759075.414420:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759075.414425:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:3.0:1787759075.414425:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759075.414431:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655251840:192.168.201.107@tcp:38:kworker.0 00000008:00100000:1.0:1787759075.414434:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759075.414437:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655251968:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759075.414451:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655252352/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787759075.414456:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787759075.414466:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787759075.414467:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759075/real 1787759075] req@ffff8c05b2d99f80 x1874588655251968/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759075.414472:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787759075.414476:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759075.414485:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12131 00000100:00080000:1.0:1787759075.414490:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787759075.414495:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655251968/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759075.414496:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759075.414502:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759075.414517:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655252480/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759075.414519:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759075.414527:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759075.414533:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655251968:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787759075.414536:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787759075.414540:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655252096:192.168.201.107@tcp:8:kworker.0 00000100:00080000:1.0:1787759075.414541:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759075.414546:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759075.414551:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12131 00000100:00080000:1.0:1787759075.414556:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759075.414560:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:3.0:1787759075.414564:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:1.0:1787759075.414566:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759075.414573:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759075/real 1787759075] req@ffff8c05b2d9bb80 x1874588655252096/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759075.414581:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655252096/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759075.414589:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655252608/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759075.414596:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759075.414603:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759075.414610:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655252096:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759075.414629:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655252224:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787759075.414642:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759075.414651:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759075/real 1787759075] req@ffff8c05b2d9b100 x1874588655252224/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759075.414661:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655252224/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759075.414668:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759075.414671:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759075.414675:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655252224:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759075.414679:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655252352:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787759075.414696:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759075.414705:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759075/real 1787759075] req@ffff8c05b2d98000 x1874588655252352/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759075.414714:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655252352/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759075.414720:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759075.414723:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759075.414726:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655252352:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759075.414731:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655252480:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787759075.414746:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759075.414755:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759075/real 1787759075] req@ffff8c05b2d98700 x1874588655252480/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759075.414782:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655252480/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759075.414790:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759075.414793:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759075.414796:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655252480:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759075.414802:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655252608:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787759075.414820:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759075.414831:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759075/real 1787759075] req@ffff8c05b2d99880 x1874588655252608/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759075.414841:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655252608/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759075.414847:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759075.414850:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759075.414852:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655252608:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759075.541927:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759075.990008:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759076.565980:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759077.014097:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759077.589965:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759078.038036:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759078.614009:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759079.062141:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759079.637956:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759080.085992:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787759080.534031:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759080.534044:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787759080.534049:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759080.534059:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12136 00000100:00080000:3.0:1787759080.534071:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759080.534089:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787759080.534134:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655252736/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759080.534161:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759080.534164:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759080.534166:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759080.534170:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12136 00000100:00080000:3.0:1787759080.534174:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759080.534179:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759080.534188:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759080.534196:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655252736:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787759080.534203:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655252864/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759080.534215:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759080.534219:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759080.534221:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759080.534224:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12136 00000100:00080000:3.0:1787759080.534228:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759080.534232:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759080.534236:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759080.534247:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655252992/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759080.534256:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759080.534259:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759080.534261:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787759080.534265:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787759080.534269:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12136 00000100:00080000:3.0:1787759080.534273:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759080.534278:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759080.534283:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759080.534296:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655253120/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759080.534307:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759080/real 1787759080] req@ffff8c05a9f7aa00 x1874588655252736/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759080.534316:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759080.534320:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759080.534322:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787759080.534325:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655252736/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759080.534328:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12136 00000100:00080000:3.0:1787759080.534335:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759080.534339:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759080.534342:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:1.0:1787759080.534345:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759080.534349:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655252736:192.168.201.107@tcp:250:kworker.0 00000008:00100000:3.0:1787759080.534350:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759080.534365:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655253248/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759080.534372:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655252864:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787759080.534378:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759080.534383:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759080.534385:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759080.534390:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12136 00000800:00000100:1.0:1787759080.534391:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787759080.534396:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787759080.534403:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759080/real 1787759080] req@ffff8c05a9f7a680 x1874588655252864/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759080.534403:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759080.534409:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759080.534414:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655252864/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759080.534421:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787759080.534424:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655253376/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759080.534425:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759080.534431:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655252864:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759080.534440:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655252992:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787759080.534445:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759080.534449:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787759080.534451:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759080.534456:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12136 00000800:00000100:1.0:1787759080.534457:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787759080.534462:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787759080.534468:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759080/real 1787759080] req@ffff8c05a9f78a80 x1874588655252992/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759080.534469:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759080.534477:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759080.534480:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655252992/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759080.534487:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787759080.534493:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655253504/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759080.534500:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759080.534504:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655252992:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787759080.534504:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759080.534512:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787759080.534514:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655253120:192.168.201.107@tcp:8:kworker.0 00000100:00080000:3.0:1787759080.534516:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759080.534522:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12136 00000100:00080000:3.0:1787759080.534527:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787759080.534531:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787759080.534532:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759080.534538:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759080.534542:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759080/real 1787759080] req@ffff8c05a9f78380 x1874588655253120/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759080.534556:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655253120/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759080.534556:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655253632/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759080.534578:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759080.534582:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759080.534585:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655253120:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759080.534596:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655253248:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759080.534607:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759080.534615:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759080/real 1787759080] req@ffff8c05a9f7b480 x1874588655253248/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759080.534625:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655253248/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759080.534630:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759080.534631:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759080.534633:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655253248:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759080.534636:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655253376:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759080.534645:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759080.534657:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759080/real 1787759080] req@ffff8c05a9f78700 x1874588655253376/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759080.534664:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655253376/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759080.534668:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759080.534670:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759080.534672:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655253376:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759080.534691:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655253504:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759080.534704:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759080.534711:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759080/real 1787759080] req@ffff8c05a9f79880 x1874588655253504/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759080.534718:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655253504/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759080.534726:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759080.534728:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759080.534730:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655253504:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759080.534734:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655253632:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759080.534743:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759080.534748:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759080/real 1787759080] req@ffff8c05a9f7a300 x1874588655253632/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759080.534754:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655253632/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759080.534759:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759080.534761:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759080.534762:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655253632:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759080.661991:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759081.110077:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759081.685999:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759082.134128:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759082.709952:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759083.158094:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759083.733975:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759084.182071:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759084.757971:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759085.206113:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787759085.653965:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759085.653975:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787759085.653979:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759085.653991:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12142 00000100:00080000:3.0:1787759085.654003:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759085.654012:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787759085.654059:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655253760/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759085.654085:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759085.654088:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759085.654089:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759085.654093:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12142 00000100:00080000:3.0:1787759085.654096:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759085.654098:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759085.654106:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759085.654115:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655253888/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759085.654122:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759085.654125:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759085.654126:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759085.654128:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12142 00000100:00080000:3.0:1787759085.654130:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759085.654135:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759085.654136:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655253760:192.168.201.107@tcp:250:kworker.0 00000008:00100000:3.0:1787759085.654138:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759085.654148:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655254016/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759085.654155:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759085.654158:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759085.654159:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759085.654162:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12142 00000100:00080000:3.0:1787759085.654164:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759085.654167:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759085.654169:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759085.654176:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655254144/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759085.654181:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759085.654183:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759085.654185:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759085.654187:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12142 00000100:00080000:3.0:1787759085.654190:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759085.654193:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759085.654195:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787759085.654199:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759085.654205:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655254272/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759085.654213:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759085.654215:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759085.654217:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759085.654220:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12142 00000100:00080000:3.0:1787759085.654221:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759085.654223:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759085.654227:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759085.654234:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655254400/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759085.654239:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787759085.654241:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759085/real 1787759085] req@ffff8c05a9f79500 x1874588655253760/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759085.654242:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787759085.654245:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759085.654251:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12142 00000100:00080000:3.0:1787759085.654256:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759085.654259:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759085.654262:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759085.654266:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655253760/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759085.654272:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655254528/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759085.654278:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759085.654283:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759085.654285:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759085.654286:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759085.654293:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787759085.654296:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12142 00000100:00100000:1.0:1787759085.654300:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655253760:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787759085.654303:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787759085.654307:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655253888:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787759085.654311:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759085.654318:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787759085.654328:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759085.654332:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655254656/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759085.654340:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759085/real 1787759085] req@ffff8c05a9f7b800 x1874588655253888/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759085.654355:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655253888/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759085.654361:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759085.654363:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759085.654365:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655253888:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759085.654385:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655254016:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759085.654402:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759085.654413:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759085/real 1787759085] req@ffff8c05a9f78e00 x1874588655254016/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759085.654424:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655254016/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759085.654432:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759085.654436:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759085.654439:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655254016:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759085.654445:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655254144:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759085.654462:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759085.654470:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759085/real 1787759085] req@ffff8c05a9f7ad80 x1874588655254144/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759085.654477:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655254144/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759085.654482:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759085.654484:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759085.654486:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655254144:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759085.654490:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655254272:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759085.654500:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759085.654505:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759085/real 1787759085] req@ffff8c05a9f7b100 x1874588655254272/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759085.654519:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655254272/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759085.654524:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759085.654526:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759085.654528:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655254272:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759085.654531:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655254400:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759085.654540:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759085.654546:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759085/real 1787759085] req@ffff8c05a9f79f80 x1874588655254400/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759085.654561:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655254400/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759085.654569:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759085.654572:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759085.654575:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655254400:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759085.654581:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655254528:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759085.654613:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759085.654623:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759085/real 1787759085] req@ffff8c05a9f7bb80 x1874588655254528/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759085.654631:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655254528/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759085.654636:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759085.654638:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759085.654640:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655254528:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759085.654644:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655254656:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759085.654655:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759085.654660:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759085/real 1787759085] req@ffff8c05a9f79180 x1874588655254656/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759085.654667:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655254656/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759085.654675:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759085.654677:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759085.654679:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655254656:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759085.781956:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759086.230042:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759086.805943:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759087.254086:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759087.829995:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759088.278054:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759088.854002:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759089.302141:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759089.878032:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759090.326047:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787759090.774068:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759090.774080:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787759090.774085:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759090.774095:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12147 00000100:00080000:0.0:1787759090.774107:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759090.774115:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787759090.774159:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588655254784/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759090.774186:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759090.774189:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787759090.774191:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759090.774195:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12147 00000100:00080000:0.0:1787759090.774198:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759090.774201:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759090.774209:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759090.774223:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588655254912/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759090.774233:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759090.774236:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787759090.774238:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759090.774244:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12147 00000100:00100000:1.0:1787759090.774244:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655254784:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787759090.774250:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759090.774255:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759090.774259:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759090.774271:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655255040/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759090.774280:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759090.774283:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787759090.774284:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759090.774288:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12147 00000100:00080000:0.0:1787759090.774305:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759090.774310:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759090.774314:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787759090.774316:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759090.774330:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655255168/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759090.774337:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759090.774340:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787759090.774341:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759090.774344:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12147 00000100:00080000:0.0:1787759090.774347:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759090.774350:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759090.774353:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759090.774360:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655255296/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759090.774365:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759090.774368:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787759090.774370:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787759090.774371:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759090/real 1787759090] req@ffff8c05a9f76a00 x1874588655254784/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759090.774374:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12147 00000100:00080000:0.0:1787759090.774377:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759090.774382:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759090.774387:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759090.774397:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655255424/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759090.774403:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759090.774405:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787759090.774406:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759090.774409:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12147 00000100:00080000:0.0:1787759090.774411:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787759090.774412:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588655254784/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759090.774415:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759090.774418:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759090.774428:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588655255552/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759090.774430:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759090.774438:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759090.774440:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787759090.774441:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787759090.774443:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759090.774448:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12147 00000100:00080000:0.0:1787759090.774451:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787759090.774451:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655254784:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787759090.774457:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759090.774461:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759090.774467:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655255680/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759090.774483:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655254912:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759090.774521:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759090.774532:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759090/real 1787759090] req@ffff8c05a9f77b80 x1874588655254912/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759090.774543:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588655254912/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759090.774556:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759090.774559:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759090.774562:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655254912:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759090.774567:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655255040:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759090.774582:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759090.774590:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759090/real 1787759090] req@ffff8c05a9f74a80 x1874588655255040/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759090.774600:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655255040/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759090.774608:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759090.774612:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759090.774615:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655255040:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759090.774621:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655255168:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759090.774638:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759090.774649:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759090/real 1787759090] req@ffff8c05a9f74380 x1874588655255168/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759090.774661:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655255168/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759090.774669:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759090.774672:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759090.774676:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655255168:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759090.774682:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655255296:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759090.774700:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759090.774710:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759090/real 1787759090] req@ffff8c05a9f76680 x1874588655255296/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759090.774720:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655255296/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759090.774728:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759090.774731:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759090.774734:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655255296:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759090.774740:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655255424:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759090.774759:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759090.774768:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759090/real 1787759090] req@ffff8c05a9f76300 x1874588655255424/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759090.774779:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655255424/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759090.774787:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759090.774790:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759090.774793:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655255424:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759090.774798:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655255552:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759090.774811:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759090.774820:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759090/real 1787759090] req@ffff8c05a9f75c00 x1874588655255552/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759090.774831:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588655255552/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759090.774838:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759090.774842:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759090.774844:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655255552:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759090.774848:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655255680:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759090.774860:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759090.774866:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759090/real 1787759090] req@ffff8c05a9f77800 x1874588655255680/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759090.774872:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655255680/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759090.774915:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759090.774919:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759090.774922:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655255680:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759090.901988:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759091.350132:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759091.925995:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759092.374129:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759092.950053:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759093.398141:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759093.974010:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759094.422097:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759094.997989:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759095.446218:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787759095.893946:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759095.893964:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787759095.893969:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759095.893978:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12152 00000100:00080000:0.0:1787759095.893988:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759095.893995:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787759095.894032:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655255808/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759095.894056:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759095.894059:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787759095.894060:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759095.894063:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12152 00000100:00080000:0.0:1787759095.894066:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759095.894070:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759095.894079:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759095.894089:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588655255936/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759095.894099:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759095.894102:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787759095.894103:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759095.894109:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12152 00000100:00080000:0.0:1787759095.894112:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787759095.894112:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655255808:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787759095.894118:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759095.894126:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759095.894139:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588655256064/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759095.894151:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759095.894155:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787759095.894158:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759095.894163:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12152 00000100:00080000:0.0:1787759095.894178:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759095.894182:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759095.894187:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787759095.894193:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759095.894201:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588655256192/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759095.894215:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759095.894219:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787759095.894221:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759095.894225:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12152 00000100:00080000:0.0:1787759095.894230:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759095.894235:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759095.894242:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759095.894242:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759095/real 1787759095] req@ffff8c05a9f75880 x1874588655255808/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759095.894261:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588655256320/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759095.894278:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759095.894281:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787759095.894283:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787759095.894288:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655255808/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759095.894289:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12152 00000100:00080000:0.0:1787759095.894295:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759095.894302:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759095.894311:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787759095.894313:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759095.894320:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759095.894323:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588655256448/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759095.894329:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655255808:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787759095.894334:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759095.894339:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787759095.894341:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759095.894344:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12152 00000100:00080000:0.0:1787759095.894347:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759095.894349:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759095.894352:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759095.894359:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588655256576/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759095.894363:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655255936:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787759095.894371:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759095.894387:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787759095.894389:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787759095.894394:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787759095.894399:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12152 00000100:00100000:1.0:1787759095.894431:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759095/real 1787759095] req@ffff8c05a9f77100 x1874588655255936/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759095.894441:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787759095.894447:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588655255936/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759095.894449:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759095.894456:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759095.894469:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655256704/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759095.894471:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759095.894482:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759095.894488:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655255936:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759095.894494:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655256064:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759095.894515:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759095.894526:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759095/real 1787759095] req@ffff8c05a9f74e00 x1874588655256064/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759095.894538:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588655256064/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759095.894553:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759095.894556:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759095.894560:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655256064:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759095.894565:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655256192:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759095.894581:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759095.894589:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759095/real 1787759095] req@ffff8c05a9f75180 x1874588655256192/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759095.894601:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588655256192/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759095.894610:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759095.894612:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759095.894615:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655256192:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759095.894621:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655256320:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759095.894636:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759095.894645:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759095/real 1787759095] req@ffff8c05a9f74000 x1874588655256320/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759095.894657:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588655256320/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759095.894666:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759095.894669:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759095.894673:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655256320:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759095.894678:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655256448:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759095.894693:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759095.894702:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759095/real 1787759095] req@ffff8c05a9f75500 x1874588655256448/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759095.894713:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588655256448/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759095.894721:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759095.894725:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759095.894728:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655256448:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759095.894746:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655256576:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759095.894761:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759095.894770:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759095/real 1787759095] req@ffff8c05a9f76d80 x1874588655256576/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759095.894782:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588655256576/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759095.894790:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759095.894794:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759095.894797:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655256576:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759095.894803:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655256704:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759095.894817:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759095.894826:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759095/real 1787759095] req@ffff8c05a9f75880 x1874588655256704/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759095.894838:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655256704/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759095.894856:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759095.894859:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759095.894863:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655256704:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759096.022022:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759096.470042:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759097.045974:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759097.494047:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759098.070009:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759098.518071:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759099.093959:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759099.542157:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759100.117995:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759100.566101:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787759101.013986:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759101.013999:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787759101.014009:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759101.014020:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12157 00000100:00080000:0.0:1787759101.014033:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759101.014042:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787759101.014084:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655256832/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759101.014110:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759101.014114:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787759101.014116:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759101.014121:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12157 00000100:00080000:0.0:1787759101.014125:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759101.014129:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787759101.014139:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655256832:192.168.201.107@tcp:250:kworker.0 00000008:00100000:0.0:1787759101.014166:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759101.014184:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588655256960/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759101.014195:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759101.014199:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787759101.014201:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759101.014205:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12157 00000100:00080000:0.0:1787759101.014209:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787759101.014214:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787759101.014217:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759101.014223:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759101.014235:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655257088/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759101.014245:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759101.014249:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787759101.014251:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759101.014255:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12157 00000100:00080000:0.0:1787759101.014259:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759101.014264:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759101.014270:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759101.014282:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655257216/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759101.014292:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759101.014295:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787759101.014297:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759101.014301:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12157 00000100:00080000:0.0:1787759101.014304:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759101.014309:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759101.014329:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759101.014341:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655257344/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787759101.014345:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759101/real 1787759101] req@ffff8c05a9f77800 x1874588655256832/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759101.014366:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759101.014371:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787759101.014372:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759101.014377:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12157 00000100:00080000:0.0:1787759101.014380:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759101.014384:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759101.014388:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759101.014399:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655257472/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759101.014407:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759101.014410:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787759101.014412:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759101.014415:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12157 00000100:00080000:0.0:1787759101.014418:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759101.014423:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759101.014427:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759101.014438:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588655257600/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759101.014447:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759101.014450:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787759101.014452:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759101.014456:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12157 00000100:00080000:0.0:1787759101.014458:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759101.014462:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759101.014467:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759101.014475:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655256832/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759101.014479:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588655257728/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759101.014499:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759101.014506:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759101.014541:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655256832:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787759101.014564:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655256960:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787759101.014586:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759101.014595:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759101/real 1787759101] req@ffff8c05a9f75c00 x1874588655256960/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759101.014603:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588655256960/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759101.014610:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759101.014613:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759101.014616:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655256960:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759101.014622:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655257088:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787759101.014640:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759101.014650:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759101/real 1787759101] req@ffff8c05a9f76300 x1874588655257088/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759101.014660:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655257088/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759101.014665:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759101.014667:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759101.014669:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655257088:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759101.014672:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655257216:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787759101.014685:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759101.014691:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759101/real 1787759101] req@ffff8c05a9f76680 x1874588655257216/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759101.014702:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655257216/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759101.014707:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759101.014709:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759101.014710:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655257216:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759101.014713:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655257344:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787759101.014722:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759101.014727:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759101/real 1787759101] req@ffff8c05a9f74380 x1874588655257344/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759101.014733:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655257344/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759101.014738:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759101.014740:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759101.014742:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655257344:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759101.014755:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655257472:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787759101.014775:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759101.014785:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759101/real 1787759101] req@ffff8c05a9f74a80 x1874588655257472/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759101.014796:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655257472/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759101.014813:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759101.014816:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759101.014818:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655257472:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759101.014821:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655257600:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787759101.014835:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759101.014842:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759101/real 1787759101] req@ffff8c05a9f77b80 x1874588655257600/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759101.014848:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588655257600/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759101.014853:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759101.014854:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759101.014856:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655257600:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759101.014859:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655257728:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787759101.014867:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759101.014872:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759101/real 1787759101] req@ffff8c05a9f76a00 x1874588655257728/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759101.014899:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588655257728/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759101.014904:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759101.014907:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759101.014910:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655257728:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759101.142006:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759101.590158:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759102.165991:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759102.614106:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759103.190007:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759103.638101:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759104.213945:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759104.662017:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759105.237977:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759105.686121:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787759106.134039:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759106.134048:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787759106.134051:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759106.134060:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12162 00000100:00080000:1.0:1787759106.134069:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759106.134075:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759106.134109:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655257856/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759106.134129:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759106.134131:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759106.134133:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759106.134135:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12162 00000100:00080000:1.0:1787759106.134138:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759106.134140:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759106.134148:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759106.134156:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588655257984/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759106.134163:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759106.134166:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759106.134167:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759106.134170:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12162 00000100:00080000:1.0:1787759106.134172:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759106.134174:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759106.134177:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759106.134185:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655257856:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787759106.134186:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655258112/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759106.134194:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759106.134196:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759106.134198:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759106.134203:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12162 00000100:00080000:1.0:1787759106.134205:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759106.134208:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759106.134210:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759106.134215:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655258240/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759106.134238:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759106.134240:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759106.134242:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759106.134246:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12162 00000100:00080000:1.0:1787759106.134249:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759106.134254:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759106.134258:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759106.134270:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655258368/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759106.134279:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787759106.134284:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787759106.134285:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759106.134290:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759106.134293:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12162 00000100:00080000:1.0:1787759106.134297:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759106.134299:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759106.134303:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759106.134311:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655258496/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759106.134317:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759106.134319:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787759106.134320:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759106.134323:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12162 00000100:00080000:1.0:1787759106.134326:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759106.134329:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759106.134332:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759106.134339:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759106/real 1787759106] req@ffff8c05b2d98380 x1874588655257856/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759106.134340:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655258624/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759106.134362:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759106.134367:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759106.134369:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759106.134370:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12162 00000100:00080000:1.0:1787759106.134373:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759106.134375:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759106.134377:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759106.134380:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655257856/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759106.134385:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655258752/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759106.134408:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759106.134414:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759106.134418:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655257856:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787759106.134426:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655257984:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787759106.134451:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759106.134476:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759106/real 1787759106] req@ffff8c05b2d9a300 x1874588655257984/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759106.134488:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588655257984/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759106.134497:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759106.134500:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759106.134504:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655257984:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759106.134529:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655258112:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787759106.134550:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759106.134560:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759106/real 1787759106] req@ffff8c05b2d99880 x1874588655258112/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759106.134572:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655258112/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759106.134579:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759106.134583:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759106.134586:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655258112:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759106.134592:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655258240:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787759106.134610:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759106.134629:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759106/real 1787759106] req@ffff8c05b2d98700 x1874588655258240/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759106.134641:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655258240/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759106.134649:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759106.134651:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759106.134654:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655258240:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759106.134660:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655258368:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787759106.134679:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759106.134690:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759106/real 1787759106] req@ffff8c05b2d98000 x1874588655258368/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759106.134701:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655258368/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759106.134709:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759106.134712:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759106.134715:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655258368:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759106.134721:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655258496:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787759106.134739:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759106.134749:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759106/real 1787759106] req@ffff8c05b2d9b100 x1874588655258496/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759106.134760:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655258496/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759106.134768:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759106.134771:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759106.134774:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655258496:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759106.134779:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655258624:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787759106.134798:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759106.134807:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759106/real 1787759106] req@ffff8c05b2d9bb80 x1874588655258624/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759106.134819:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655258624/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759106.134827:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759106.134830:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759106.134833:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655258624:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759106.134838:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655258752:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787759106.134856:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759106.134866:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759106/real 1787759106] req@ffff8c05b2d99f80 x1874588655258752/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759106.134902:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655258752/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759106.134911:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759106.134915:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759106.134918:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655258752:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759106.261968:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759106.710046:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759107.285991:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759107.734145:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759108.309948:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759108.758042:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759109.334088:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759109.782085:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759110.358017:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759110.806081:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787759111.254019:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759111.254031:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787759111.254036:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759111.254047:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12167 00000100:00080000:3.0:1787759111.254059:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759111.254068:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787759111.254110:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655258880/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759111.254161:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759111.254165:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759111.254167:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759111.254172:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12167 00000100:00080000:3.0:1787759111.254175:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759111.254180:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759111.254189:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759111.254204:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655258880:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787759111.254205:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655259008/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759111.254221:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759111.254224:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759111.254226:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759111.254231:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12167 00000100:00080000:3.0:1787759111.254234:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759111.254238:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759111.254243:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759111.254268:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655259136/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759111.254278:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759111.254282:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759111.254284:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787759111.254290:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787759111.254292:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12167 00000100:00080000:3.0:1787759111.254297:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759111.254302:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759111.254307:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759111.254319:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655259264/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759111.254329:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759111.254332:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759111.254334:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759111.254338:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12167 00000100:00100000:1.0:1787759111.254343:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759111/real 1787759111] req@ffff8c05a9f79180 x1874588655258880/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759111.254343:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759111.254350:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759111.254358:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759111.254370:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655258880/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759111.254373:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655259392/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759111.254400:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759111.254407:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787759111.254408:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787759111.254414:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655258880:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787759111.254414:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759111.254416:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759111.254428:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12167 00000100:00080000:3.0:1787759111.254433:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759111.254437:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759111.254442:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759111.254450:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655259008:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759111.254456:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655259520/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759111.254468:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759111.254472:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787759111.254474:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759111.254480:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12167 00000800:00000100:1.0:1787759111.254481:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787759111.254485:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759111.254490:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759111.254495:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759111.254500:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759111/real 1787759111] req@ffff8c05a9f7bb80 x1874588655259008/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759111.254507:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655259648/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759111.254523:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787759111.254526:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655259008/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759111.254538:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759111.254540:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759111.254541:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759111.254548:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787759111.254550:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12167 00000100:00100000:1.0:1787759111.254557:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655259008:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787759111.254557:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759111.254565:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759111.254569:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655259136:192.168.201.107@tcp:38:kworker.0 00000008:00100000:3.0:1787759111.254572:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759111.254588:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655259776/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787759111.254598:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759111.254614:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759111/real 1787759111] req@ffff8c05a9f79f80 x1874588655259136/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759111.254628:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655259136/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759111.254637:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759111.254640:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759111.254644:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655259136:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759111.254650:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655259264:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759111.254669:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759111.254679:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759111/real 1787759111] req@ffff8c05a9f7b100 x1874588655259264/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759111.254690:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655259264/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759111.254698:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759111.254702:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759111.254705:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655259264:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759111.254710:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655259392:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759111.254727:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759111.254737:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759111/real 1787759111] req@ffff8c05a9f7ad80 x1874588655259392/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759111.254748:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655259392/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759111.254756:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759111.254759:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759111.254762:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655259392:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759111.254778:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655259520:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759111.254796:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759111.254806:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759111/real 1787759111] req@ffff8c05a9f78e00 x1874588655259520/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759111.254825:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655259520/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759111.254833:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759111.254836:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759111.254839:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655259520:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759111.254844:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655259648:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759111.254859:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759111.254868:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759111/real 1787759111] req@ffff8c05a9f7b800 x1874588655259648/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759111.254901:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655259648/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759111.254909:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759111.254912:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759111.254916:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655259648:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759111.254921:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655259776:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759111.254938:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759111.254947:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759111/real 1787759111] req@ffff8c05a9f79500 x1874588655259776/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759111.254957:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655259776/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759111.254971:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759111.254974:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759111.254977:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655259776:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759111.381952:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759111.830164:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759112.405979:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759112.854101:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759113.429958:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759113.878129:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759114.453967:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759114.902016:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759115.477971:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759115.926123:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787759116.373995:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759116.374006:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787759116.374011:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759116.374021:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12172 00000100:00080000:3.0:1787759116.374032:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759116.374040:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787759116.374089:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655259904/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759116.374121:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759116.374124:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759116.374127:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759116.374131:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12172 00000100:00080000:3.0:1787759116.374136:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759116.374140:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759116.374150:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759116.374162:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655260032/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759116.374166:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655259904:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787759116.374172:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759116.374175:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759116.374177:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759116.374180:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12172 00000100:00080000:3.0:1787759116.374183:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759116.374186:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759116.374190:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759116.374198:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655260160/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759116.374206:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759116.374208:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759116.374209:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759116.374212:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12172 00000100:00080000:3.0:1787759116.374214:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759116.374217:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759116.374221:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759116.374228:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655260288/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787759116.374230:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787759116.374235:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759116.374238:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759116.374239:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759116.374242:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12172 00000100:00080000:3.0:1787759116.374244:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759116.374247:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759116.374250:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759116.374261:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655260416/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759116.374272:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759116/real 1787759116] req@ffff8c05a9f7a300 x1874588655259904/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759116.374273:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759116.374277:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759116.374278:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759116.374281:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12172 00000100:00080000:3.0:1787759116.374284:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759116.374287:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759116.374289:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655259904/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787759116.374291:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787759116.374299:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787759116.374299:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655260544/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759116.374303:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759116.374306:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655259904:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787759116.374307:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759116.374310:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787759116.374311:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759116.374314:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12172 00000100:00080000:3.0:1787759116.374316:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759116.374319:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759116.374323:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759116.374327:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655260032:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759116.374330:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655260672/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759116.374338:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759116.374341:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759116.374343:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787759116.374345:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787759116.374346:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12172 00000100:00080000:3.0:1787759116.374349:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759116.374352:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759116.374355:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759116/real 1787759116] req@ffff8c05a9f79880 x1874588655260032/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787759116.374356:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759116.374365:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655260800/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759116.374367:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655260032/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759116.374380:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759116.374383:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759116.374386:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655260032:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759116.374391:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655260160:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759116.374409:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759116.374419:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759116/real 1787759116] req@ffff8c05a9f78700 x1874588655260160/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759116.374426:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655260160/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759116.374431:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759116.374433:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759116.374435:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655260160:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759116.374438:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655260288:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759116.374451:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759116.374457:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759116/real 1787759116] req@ffff8c05a9f7b480 x1874588655260288/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759116.374463:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655260288/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759116.374468:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759116.374469:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759116.374472:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655260288:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759116.374475:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655260416:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759116.374484:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759116.374489:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759116/real 1787759116] req@ffff8c05a9f78380 x1874588655260416/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759116.374495:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655260416/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759116.374499:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759116.374501:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759116.374502:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655260416:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759116.374505:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655260544:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759116.374513:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759116.374518:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759116/real 1787759116] req@ffff8c05a9f78a80 x1874588655260544/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759116.374524:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655260544/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759116.374529:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759116.374530:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759116.374532:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655260544:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759116.374542:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655260672:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759116.374551:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759116.374556:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759116/real 1787759116] req@ffff8c05a9f7a680 x1874588655260672/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759116.374563:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655260672/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759116.374567:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759116.374569:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759116.374570:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655260672:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759116.374573:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655260800:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759116.374582:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759116.374587:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759116/real 1787759116] req@ffff8c05a9f7aa00 x1874588655260800/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759116.374593:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655260800/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759116.374597:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759116.374599:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759116.374600:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655260800:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759116.501945:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759116.949978:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759117.525973:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759117.974156:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759118.549938:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759118.998035:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759119.574024:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759120.022118:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759120.597983:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759121.046123:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787759121.493967:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759121.493976:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787759121.493980:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759121.493988:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12177 00000100:00080000:3.0:1787759121.493998:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759121.494004:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787759121.494038:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655260928/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759121.494061:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759121.494063:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759121.494064:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759121.494067:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12177 00000100:00080000:3.0:1787759121.494069:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759121.494071:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759121.494079:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759121.494087:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655261056/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759121.494094:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759121.494102:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759121.494104:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759121.494108:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12177 00000100:00100000:1.0:1787759121.494111:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655260928:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787759121.494111:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759121.494117:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759121.494122:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759121.494129:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655261184/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759121.494135:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759121.494138:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759121.494139:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759121.494142:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12177 00000100:00080000:3.0:1787759121.494144:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759121.494146:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759121.494149:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759121.494156:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655261312/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759121.494162:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759121.494165:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759121.494166:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759121.494172:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12177 00000800:00000100:1.0:1787759121.494173:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787759121.494179:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759121.494184:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759121.494189:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759121.494202:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655261440/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759121.494211:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787759121.494213:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759121/real 1787759121] req@ffff8c05a9f7aa00 x1874588655260928/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759121.494215:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759121.494220:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759121.494229:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12177 00000100:00080000:3.0:1787759121.494236:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759121.494241:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759121.494244:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655260928/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787759121.494247:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787759121.494261:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787759121.494262:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655261568/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759121.494271:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787759121.494272:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759121.494276:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787759121.494278:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787759121.494280:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655260928:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787759121.494283:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12177 00000100:00080000:3.0:1787759121.494288:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759121.494292:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759121.494295:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655261056:192.168.201.107@tcp:38:kworker.0 00000008:00100000:3.0:1787759121.494296:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759121.494308:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655261696/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759121.494314:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759121.494316:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759121.494317:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787759121.494321:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787759121.494322:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12177 00000100:00080000:3.0:1787759121.494327:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759121.494330:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759121.494334:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759121.494337:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759121/real 1787759121] req@ffff8c05a9f7a680 x1874588655261056/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759121.494347:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655261056/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759121.494350:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655261824/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759121.494360:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759121.494365:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759121.494369:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655261056:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759121.494395:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655261184:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759121.494416:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759121.494427:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759121/real 1787759121] req@ffff8c05a9f78a80 x1874588655261184/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759121.494435:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655261184/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759121.494443:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759121.494445:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759121.494447:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655261184:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759121.494450:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655261312:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759121.494462:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759121.494480:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759121/real 1787759121] req@ffff8c05a9f78380 x1874588655261312/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759121.494487:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655261312/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759121.494492:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759121.494494:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759121.494495:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655261312:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759121.494499:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655261440:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759121.494509:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759121.494515:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759121/real 1787759121] req@ffff8c05a9f7b480 x1874588655261440/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759121.494521:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655261440/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759121.494525:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759121.494527:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759121.494529:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655261440:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759121.494532:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655261568:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759121.494541:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759121.494546:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759121/real 1787759121] req@ffff8c05a9f78700 x1874588655261568/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759121.494552:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655261568/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759121.494556:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759121.494558:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759121.494559:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655261568:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759121.494562:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655261696:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759121.494570:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759121.494575:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759121/real 1787759121] req@ffff8c05a9f79880 x1874588655261696/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759121.494581:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655261696/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759121.494586:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759121.494587:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759121.494589:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655261696:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759121.494591:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655261824:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759121.494599:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759121.494604:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759121/real 1787759121] req@ffff8c05a9f7a300 x1874588655261824/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759121.494611:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655261824/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759121.494616:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759121.494617:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759121.494618:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655261824:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759121.622017:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759122.070117:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759122.646030:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759123.094070:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759123.669974:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759124.118104:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759124.694015:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759125.142060:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759125.717948:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759126.166096:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787759126.613949:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759126.613958:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787759126.613962:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759126.613971:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12183 00000100:00080000:3.0:1787759126.613981:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759126.613987:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787759126.614022:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655261952/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759126.614049:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759126.614052:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759126.614053:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759126.614056:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12183 00000100:00080000:3.0:1787759126.614058:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759126.614060:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759126.614068:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759126.614077:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655262080/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759126.614083:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759126.614086:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759126.614087:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787759126.614092:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655261952:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787759126.614093:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12183 00000100:00080000:3.0:1787759126.614095:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759126.614101:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759126.614104:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759126.614112:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655262208/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759126.614118:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759126.614120:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759126.614121:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759126.614124:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12183 00000100:00080000:3.0:1787759126.614127:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759126.614130:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759126.614133:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759126.614140:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655262336/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759126.614145:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759126.614148:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759126.614149:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759126.614151:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12183 00000100:00080000:3.0:1787759126.614154:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759126.614156:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759126.614159:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787759126.614161:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759126.614170:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655262464/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759126.614176:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759126.614179:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759126.614180:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759126.614183:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12183 00000100:00080000:3.0:1787759126.614185:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759126.614187:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759126.614190:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759126.614198:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655262592/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759126.614204:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759126/real 1787759126] req@ffff8c05a9f79500 x1874588655261952/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759126.614223:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759126.614227:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787759126.614228:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759126.614232:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12183 00000100:00100000:1.0:1787759126.614234:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655261952/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759126.614236:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759126.614244:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:1.0:1787759126.614247:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787759126.614250:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787759126.614252:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759126.614257:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655261952:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787759126.614275:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655262720/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759126.614276:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655262080:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787759126.614291:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759126.614295:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1787759126.614298:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787759126.614300:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759126.614306:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12183 00000100:00100000:1.0:1787759126.614309:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759126/real 1787759126] req@ffff8c05a9f7b800 x1874588655262080/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759126.614314:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759126.614322:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759126.614330:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759126.614332:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655262080/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759126.614339:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759126.614343:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759126.614344:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655262848/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759126.614347:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655262080:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759126.614356:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655262208:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759126.614371:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759126.614379:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759126/real 1787759126] req@ffff8c05a9f78e00 x1874588655262208/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759126.614386:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655262208/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759126.614390:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759126.614392:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759126.614394:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655262208:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759126.614397:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655262336:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759126.614406:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759126.614411:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759126/real 1787759126] req@ffff8c05a9f7ad80 x1874588655262336/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759126.614417:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655262336/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759126.614425:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759126.614427:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759126.614428:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655262336:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759126.614431:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655262464:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759126.614440:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759126.614444:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759126/real 1787759126] req@ffff8c05a9f7b100 x1874588655262464/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759126.614450:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655262464/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759126.614454:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759126.614456:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759126.614458:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655262464:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759126.614460:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655262592:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759126.614469:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759126.614473:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759126/real 1787759126] req@ffff8c05a9f79f80 x1874588655262592/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759126.614480:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655262592/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759126.614484:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759126.614485:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759126.614487:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655262592:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759126.614497:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655262720:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759126.614507:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759126.614512:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759126/real 1787759126] req@ffff8c05a9f7bb80 x1874588655262720/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759126.614518:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655262720/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759126.614522:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759126.614524:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759126.614525:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655262720:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759126.614528:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655262848:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759126.614536:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759126.614541:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759126/real 1787759126] req@ffff8c05a9f79180 x1874588655262848/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759126.614547:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655262848/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759126.614551:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759126.614553:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759126.614554:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655262848:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759126.741960:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759127.190055:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759127.766004:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759128.214132:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759128.789948:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759129.238063:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759129.813995:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759130.262113:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759130.838025:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759131.286059:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787759131.733998:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759131.734009:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787759131.734014:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759131.734024:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12188 00000100:00080000:0.0:1787759131.734034:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759131.734040:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787759131.734076:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588655262976/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759131.734097:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759131.734100:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787759131.734101:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759131.734104:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12188 00000100:00080000:0.0:1787759131.734106:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759131.734109:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759131.734117:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759131.734126:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588655263104/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759131.734133:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655262976:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787759131.734135:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759131.734141:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787759131.734142:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759131.734145:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12188 00000100:00080000:0.0:1787759131.734146:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759131.734148:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759131.734151:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759131.734159:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655263232/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759131.734166:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759131.734169:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787759131.734170:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759131.734172:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12188 00000100:00080000:0.0:1787759131.734187:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759131.734189:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759131.734192:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759131.734199:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655263360/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759131.734204:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759131.734207:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787759131.734208:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759131.734210:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12188 00000100:00080000:0.0:1787759131.734212:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759131.734214:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759131.734217:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759131.734224:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655263488/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759131.734230:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759131.734233:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787759131.734234:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759131.734237:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12188 00000100:00080000:0.0:1787759131.734241:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787759131.734244:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655263104:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787759131.734246:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759131.734252:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759131.734262:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655263616/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759131.734268:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759131.734271:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787759131.734272:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759131.734275:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12188 00000100:00080000:0.0:1787759131.734277:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759131.734279:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759131.734283:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759131.734286:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655263232:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759131.734293:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588655263744/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759131.734299:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759131.734301:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787759131.734302:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759131.734304:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12188 00000100:00080000:0.0:1787759131.734306:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759131.734307:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759131.734310:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759131.734317:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655263872/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759131.734326:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655263360:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759131.734362:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655263488:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759131.734392:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655263616:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759131.734413:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655263744:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759131.734433:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655263872:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759131.861961:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759132.310067:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759132.886041:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759133.334115:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759133.909975:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759134.358063:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787759134.806142:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.107:988 00000400:00000100:2.0:1787759134.806175:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.107@tcp at host 192.168.201.107 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787759134.806190:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787759134.806222:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787759134.806228:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787759134.806236:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787759134.806247:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787759134.806256:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787759134.806265:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787759134.806274:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:1.0:1787759134.806281:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759131/real 1787759134] req@ffff8c05a9f77b80 x1874588655263104/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759134.806308:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588655263104/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759134.806321:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759134.806326:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759134.806331:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655263104:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759134.806339:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759131/real 1787759134] req@ffff8c05a9f74a80 x1874588655263232/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759134.806350:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655263232/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759134.806358:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759134.806362:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759134.806365:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655263232:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759134.806371:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759131/real 1787759134] req@ffff8c05a9f74380 x1874588655263360/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759134.806382:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655263360/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759134.806389:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759134.806393:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759134.806396:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655263360:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759134.806402:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759131/real 1787759134] req@ffff8c05a9f76680 x1874588655263488/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759134.806413:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655263488/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759134.806421:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759134.806424:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759134.806427:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655263488:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759134.806432:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759131/real 1787759134] req@ffff8c05a9f76a00 x1874588655262976/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759134.806444:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588655262976/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759134.806452:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759134.806455:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759134.806458:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655262976:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787759134.806483:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759131/real 1787759134] req@ffff8c05a9f76300 x1874588655263616/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759134.806494:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655263616/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759134.806501:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759134.806504:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759134.806507:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655263616:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759134.806513:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759131/real 1787759134] req@ffff8c05a9f75c00 x1874588655263744/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759134.806524:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588655263744/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759134.806532:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759134.806534:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759134.806544:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655263744:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759134.806550:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759131/real 1787759134] req@ffff8c05a9f77800 x1874588655263872/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759134.806562:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655263872/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759134.806569:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759134.806572:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759134.806576:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655263872:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759134.933975:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759135.382114:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759135.957957:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759136.406130:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787759136.854082:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759136.854092:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787759136.854096:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759136.854107:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12193 00000100:00080000:2.0:1787759136.854126:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759136.854135:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759136.854170:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588655264000/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759136.854202:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759136.854206:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759136.854209:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759136.854214:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12193 00000100:00080000:2.0:1787759136.854219:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759136.854224:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759136.854233:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759136.854250:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588655264128/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759136.854259:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759136.854262:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759136.854264:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759136.854267:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12193 00000100:00080000:2.0:1787759136.854271:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759136.854276:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759136.854281:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759136.854291:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588655264256/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759136.854301:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759136.854304:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759136.854312:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759136.854318:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12193 00000100:00080000:2.0:1787759136.854322:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759136.854327:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759136.854332:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759136.854346:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588655264384/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759136.854358:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787759136.854364:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655264000:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787759136.854364:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759136.854367:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759136.854376:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12193 00000100:00080000:2.0:1787759136.854381:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759136.854392:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759136.854398:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759136.854411:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588655264512/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759136.854420:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759136.854423:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759136.854424:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759136.854428:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12193 00000100:00080000:2.0:1787759136.854432:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759136.854436:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759136.854441:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787759136.854455:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759136.854455:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588655264640/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759136.854467:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759136.854471:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787759136.854473:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759136.854478:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12193 00000100:00080000:2.0:1787759136.854483:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759136.854487:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759136.854492:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759136.854503:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588655264768/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759136.854511:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759136.854514:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759136.854519:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787759136.854520:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759136/real 1787759136] req@ffff8c05b2e73100 x1874588655264000/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759136.854526:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12193 00000100:00080000:2.0:1787759136.854534:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759136.854540:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759136.854546:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759136.854551:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588655264000/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759136.854561:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588655264896/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759136.854565:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759136.854575:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759136.854582:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655264000:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787759136.854592:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655264128:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759136.854619:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759136.854633:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759136/real 1787759136] req@ffff8c05b2e70000 x1874588655264128/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759136.854721:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588655264128/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759136.854731:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759136.854735:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759136.854738:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655264128:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759136.854743:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655264256:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759136.854767:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759136.854779:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759136/real 1787759136] req@ffff8c05b2e70a80 x1874588655264256/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759136.854789:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588655264256/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759136.854796:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759136.854799:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759136.854801:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655264256:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759136.854831:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655264384:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759136.854849:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759136.854859:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759136/real 1787759136] req@ffff8c05b2e72300 x1874588655264384/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759136.854868:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588655264384/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759136.854905:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759136.854910:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759136.854913:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655264384:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759136.854920:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655264512:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759136.854938:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759136.854951:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759136/real 1787759136] req@ffff8c05b2e72a00 x1874588655264512/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759136.854966:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588655264512/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759136.854973:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759136.854975:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759136.854978:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655264512:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759136.854983:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655264640:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759136.854999:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759136.855008:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759136/real 1787759136] req@ffff8c05b2e72680 x1874588655264640/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759136.855020:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588655264640/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759136.855028:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759136.855031:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759136.855034:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655264640:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759136.855040:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655264768:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759136.855057:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759136.855067:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759136/real 1787759136] req@ffff8c05b2e71180 x1874588655264768/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759136.855076:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588655264768/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759136.855088:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759136.855091:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759136.855094:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655264768:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759136.855099:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655264896:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759136.855117:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759136.855126:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759136/real 1787759136] req@ffff8c05b2e72d80 x1874588655264896/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759136.855136:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588655264896/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759136.855143:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759136.855146:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759136.855149:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655264896:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759136.981983:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759137.430096:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759138.006018:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759138.454112:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759139.030001:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759139.478063:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759140.054001:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759140.502160:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759141.078007:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759141.526078:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787759141.973979:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759141.973998:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787759141.974003:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759141.974015:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12198 00000100:00080000:2.0:1787759141.974027:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759141.974037:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759141.974081:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588655265024/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759141.974113:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759141.974117:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759141.974120:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759141.974126:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12198 00000100:00080000:2.0:1787759141.974131:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759141.974137:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787759141.974146:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655265024:192.168.201.107@tcp:250:kworker.0 00000008:00100000:2.0:1787759141.974148:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759141.974168:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588655265152/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759141.974182:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759141.974186:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759141.974189:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759141.974193:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12198 00000100:00080000:2.0:1787759141.974197:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759141.974202:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759141.974209:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787759141.974212:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759141.974228:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588655265280/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759141.974241:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759141.974244:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759141.974247:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759141.974251:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12198 00000100:00080000:2.0:1787759141.974255:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759141.974260:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759141.974265:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759141.974279:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759141/real 1787759141] req@ffff8c05b2e70380 x1874588655265024/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759141.974281:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588655265408/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759141.974307:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787759141.974311:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588655265024/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759141.974314:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759141.974320:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759141.974326:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759141.974327:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12198 00000100:00080000:0.0:1787759141.974334:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787759141.974336:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787759141.974339:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655265024:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787759141.974346:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759141.974354:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759141.974369:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655265152:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759141.974369:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588655265536/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759141.974389:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759141.974393:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759141.974396:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759141.974401:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12198 00000800:00000100:0.0:1787759141.974405:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787759141.974409:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759141.974417:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787759141.974420:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759141/real 1787759141] req@ffff8c05b2e70700 x1874588655265152/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787759141.974425:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759141.974434:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588655265152/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759141.974442:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787759141.974446:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588655265664/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759141.974447:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759141.974453:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655265152:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759141.974464:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655265280:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787759141.974472:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759141.974477:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787759141.974480:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787759141.974485:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787759141.974489:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12198 00000100:00080000:2.0:1787759141.974495:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787759141.974496:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759141/real 1787759141] req@ffff8c05b2e73480 x1874588655265280/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759141.974505:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787759141.974511:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588655265280/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787759141.974512:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787759141.974524:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759141.974527:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759141.974532:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655265280:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759141.974534:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588655265792/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759141.974539:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655265408:192.168.201.107@tcp:8:kworker.0 00000100:00080000:2.0:1787759141.974554:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759141.974559:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1787759141.974561:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787759141.974563:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787759141.974571:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759141/real 1787759141] req@ffff8c05b2e71500 x1874588655265408/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759141.974573:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12198 00000100:00080000:2.0:1787759141.974581:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787759141.974587:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588655265408/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759141.974589:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759141.974615:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787759141.974618:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759141.974624:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759141.974628:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655265408:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759141.974640:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588655265920/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759141.974649:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655265536:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759141.974673:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759141.974689:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759141/real 1787759141] req@ffff8c05b2e73b80 x1874588655265536/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759141.974698:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588655265536/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759141.974703:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759141.974705:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759141.974707:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655265536:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759141.974711:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655265664:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759141.974723:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759141.974729:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759141/real 1787759141] req@ffff8c05b2e71c00 x1874588655265664/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759141.974736:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588655265664/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759141.974746:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759141.974748:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759141.974750:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655265664:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759141.974753:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655265792:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759141.974763:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759141.974768:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759141/real 1787759141] req@ffff8c05b2e71880 x1874588655265792/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759141.974775:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588655265792/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759141.974780:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759141.974782:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759141.974784:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655265792:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759141.974797:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655265920:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759141.974817:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759141.974828:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759141/real 1787759141] req@ffff8c05b2e70e00 x1874588655265920/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759141.974841:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588655265920/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759141.974847:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759141.974849:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759141.974852:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655265920:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759142.102024:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759142.550118:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759143.125970:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759143.574084:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759144.150024:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759144.598165:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759145.174040:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759145.622056:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759146.198023:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759146.646173:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787759147.094030:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759147.094039:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787759147.094043:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759147.094052:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12203 00000100:00080000:0.0:1787759147.094064:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759147.094072:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787759147.094116:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655266048/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759147.094143:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759147.094146:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787759147.094147:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759147.094151:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12203 00000100:00080000:0.0:1787759147.094154:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759147.094157:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759147.094165:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759147.094180:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588655266176/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759147.094191:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759147.094194:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787759147.094197:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759147.094201:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12203 00000100:00080000:0.0:1787759147.094215:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759147.094220:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759147.094225:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759147.094230:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655266048:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787759147.094238:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588655266304/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759147.094248:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759147.094251:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787759147.094253:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759147.094255:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12203 00000100:00080000:0.0:1787759147.094258:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759147.094261:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759147.094265:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759147.094274:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588655266432/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759147.094280:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759147.094283:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787759147.094284:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759147.094286:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12203 00000100:00080000:0.0:1787759147.094288:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759147.094291:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:1.0:1787759147.094295:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:0.0:1787759147.094296:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759147.094307:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588655266560/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759147.094323:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759147.094326:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787759147.094328:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759147.094333:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12203 00000100:00080000:0.0:1787759147.094338:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787759147.094342:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759147/real 1787759147] req@ffff8c05a9f75880 x1874588655266048/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759147.094346:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759147.094357:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759147.094365:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655266048/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759147.094373:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588655266688/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759147.094376:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759147.094384:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787759147.094389:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787759147.094391:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655266048:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787759147.094394:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787759147.094399:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787759147.094402:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655266176:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787759147.094403:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12203 00000800:00000100:1.0:1787759147.094420:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787759147.094421:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759147.094429:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759147.094432:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759147/real 1787759147] req@ffff8c05a9f76d80 x1874588655266176/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787759147.094438:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759147.094453:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588655266176/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759147.094461:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588655266816/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759147.094462:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759147.094469:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759147.094476:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655266176:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787759147.094484:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759147.094487:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787759147.094489:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759147.094493:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12203 00000100:00080000:0.0:1787759147.094497:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759147.094518:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759147.094523:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655266304:192.168.201.107@tcp:38:kworker.0 00000008:00100000:0.0:1787759147.094527:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759147.094547:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588655266944/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787759147.094548:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759147.094561:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759147/real 1787759147] req@ffff8c05a9f75500 x1874588655266304/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759147.094579:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588655266304/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759147.094586:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759147.094588:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759147.094591:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655266304:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759147.094595:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655266432:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759147.094607:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759147.094614:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759147/real 1787759147] req@ffff8c05a9f74000 x1874588655266432/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759147.094621:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588655266432/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759147.094631:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759147.094633:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759147.094635:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655266432:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759147.094638:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655266560:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759147.094648:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759147.094653:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759147/real 1787759147] req@ffff8c05a9f75180 x1874588655266560/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759147.094660:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588655266560/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759147.094666:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759147.094668:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759147.094670:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655266560:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759147.094673:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655266688:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759147.094683:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759147.094688:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759147/real 1787759147] req@ffff8c05a9f74e00 x1874588655266688/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759147.094696:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588655266688/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759147.094701:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759147.094702:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759147.094705:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655266688:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759147.094708:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655266816:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759147.094718:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759147.094724:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759147/real 1787759147] req@ffff8c05a9f77100 x1874588655266816/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759147.094731:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588655266816/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759147.094736:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759147.094738:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759147.094740:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655266816:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759147.094751:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655266944:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759147.094762:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759147.094775:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759147/real 1787759147] req@ffff8c05a9f76d80 x1874588655266944/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759147.094784:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588655266944/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759147.094789:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759147.094791:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759147.094793:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655266944:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759147.221962:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759147.670037:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759148.245998:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759148.694193:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759149.270001:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759149.718061:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759150.293999:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759150.742122:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759151.318023:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759151.766002:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787759152.213983:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759152.213994:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787759152.213998:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759152.214009:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12208 00000100:00080000:3.0:1787759152.214020:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759152.214026:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787759152.214073:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655267072/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759152.214101:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759152.214103:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759152.214106:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759152.214109:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12208 00000100:00080000:3.0:1787759152.214112:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759152.214116:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759152.214124:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759152.214134:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655267200/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759152.214142:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787759152.214145:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655267072:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787759152.214145:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759152.214147:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759152.214154:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12208 00000100:00080000:3.0:1787759152.214157:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759152.214159:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759152.214162:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759152.214170:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655267328/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759152.214176:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759152.214179:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759152.214180:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759152.214183:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12208 00000100:00080000:3.0:1787759152.214185:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759152.214187:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759152.214189:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759152.214196:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655267456/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759152.214202:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759152.214204:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759152.214205:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759152.214207:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12208 00000100:00080000:3.0:1787759152.214209:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759152.214211:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759152.214214:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759152.214220:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655267584/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759152.214226:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787759152.214227:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787759152.214228:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759152.214231:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759152.214234:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12208 00000100:00080000:3.0:1787759152.214236:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759152.214247:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759152.214250:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759152.214259:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655267712/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759152.214264:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759152.214266:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787759152.214267:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759152.214271:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12208 00000100:00080000:3.0:1787759152.214273:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759152.214275:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759152.214278:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759152.214286:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759152/real 1787759152] req@ffff8c05a9f79180 x1874588655267072/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759152.214287:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655267840/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759152.214305:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759152.214311:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759152.214312:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759152.214314:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12208 00000100:00080000:3.0:1787759152.214317:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759152.214319:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759152.214322:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759152.214325:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655267072/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759152.214330:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655267968/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759152.214348:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759152.214354:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759152.214360:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655267072:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787759152.214390:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655267200:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759152.214429:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759152.214442:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759152/real 1787759152] req@ffff8c05a9f7bb80 x1874588655267200/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759152.214456:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655267200/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759152.214465:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759152.214468:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759152.214472:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655267200:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759152.214478:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655267328:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759152.214496:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759152.214506:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759152/real 1787759152] req@ffff8c05a9f79f80 x1874588655267328/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759152.214525:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655267328/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759152.214533:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759152.214536:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759152.214539:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655267328:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759152.214545:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655267456:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759152.214562:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759152.214571:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759152/real 1787759152] req@ffff8c05a9f7b100 x1874588655267456/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759152.214584:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655267456/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759152.214591:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759152.214595:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759152.214598:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655267456:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759152.214604:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655267584:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759152.214625:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759152.214642:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759152/real 1787759152] req@ffff8c05a9f7ad80 x1874588655267584/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759152.214654:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655267584/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759152.214667:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759152.214669:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759152.214672:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655267584:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759152.214678:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655267712:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759152.214697:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759152.214705:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759152/real 1787759152] req@ffff8c05a9f78e00 x1874588655267712/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759152.214715:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655267712/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759152.214722:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759152.214725:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759152.214728:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655267712:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759152.214733:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655267840:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759152.214749:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759152.214757:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759152/real 1787759152] req@ffff8c05a9f7b800 x1874588655267840/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759152.214767:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655267840/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759152.214774:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759152.214777:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759152.214781:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655267840:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759152.214786:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655267968:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759152.214800:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759152.214810:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759152/real 1787759152] req@ffff8c05a9f79500 x1874588655267968/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759152.214820:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655267968/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759152.214828:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759152.214831:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759152.214834:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655267968:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759152.342109:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759152.790161:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759153.365948:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759153.814006:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759154.389986:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759154.838111:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759155.414215:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759155.862104:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759156.437990:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759156.886160:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787759157.334002:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759157.334011:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787759157.334015:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759157.334024:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12213 00000100:00080000:3.0:1787759157.334035:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759157.334041:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787759157.334076:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655268096/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759157.334105:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759157.334107:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759157.334109:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759157.334112:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12213 00000100:00080000:3.0:1787759157.334114:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759157.334116:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759157.334123:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759157.334131:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655268224/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759157.334137:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759157.334139:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759157.334140:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759157.334142:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12213 00000100:00080000:3.0:1787759157.334144:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759157.334146:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759157.334148:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759157.334153:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655268352/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759157.334158:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759157.334160:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759157.334161:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759157.334164:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12213 00000100:00080000:3.0:1787759157.334166:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759157.334168:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759157.334171:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759157.334175:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655268480/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759157.334180:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759157.334182:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759157.334184:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759157.334186:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12213 00000100:00080000:3.0:1787759157.334188:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759157.334191:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759157.334193:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759157.334200:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655268608/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759157.334205:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759157.334207:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759157.334208:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759157.334211:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12213 00000100:00080000:3.0:1787759157.334212:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759157.334215:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759157.334217:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759157.334222:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655268736/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759157.334230:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759157.334232:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787759157.334233:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759157.334236:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12213 00000100:00080000:3.0:1787759157.334239:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759157.334245:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759157.334251:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655268096:192.168.201.107@tcp:250:kworker.0 00000008:00100000:3.0:1787759157.334252:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759157.334270:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655268864/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759157.334281:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759157.334283:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759157.334286:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759157.334289:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12213 00000100:00080000:3.0:1787759157.334292:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759157.334296:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759157.334300:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759157.334309:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655268992/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787759157.334359:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759157.334408:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759157/real 1787759157] req@ffff8c05a9f7a300 x1874588655268096/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759157.334428:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655268096/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759157.334439:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759157.334444:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759157.334447:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655268096:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787759157.334458:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655268224:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759157.334481:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759157.334494:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759157/real 1787759157] req@ffff8c05a9f79880 x1874588655268224/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759157.334505:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655268224/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759157.334512:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759157.334515:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759157.334518:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655268224:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759157.334524:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655268352:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759157.334542:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759157.334549:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759157/real 1787759157] req@ffff8c05a9f78700 x1874588655268352/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759157.334556:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655268352/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759157.334561:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759157.334563:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759157.334565:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655268352:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759157.334568:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655268480:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759157.334580:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759157.334585:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759157/real 1787759157] req@ffff8c05a9f7b480 x1874588655268480/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759157.334592:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655268480/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759157.334597:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759157.334598:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759157.334600:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655268480:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759157.334603:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655268608:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759157.334613:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759157.334618:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759157/real 1787759157] req@ffff8c05a9f78380 x1874588655268608/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759157.334624:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655268608/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759157.334629:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759157.334630:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759157.334632:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655268608:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759157.334654:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655268736:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759157.334664:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759157.334669:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759157/real 1787759157] req@ffff8c05a9f78a80 x1874588655268736/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759157.334675:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655268736/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759157.334683:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759157.334685:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759157.334687:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655268736:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759157.334690:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655268864:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759157.334699:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759157.334704:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759157/real 1787759157] req@ffff8c05a9f7a680 x1874588655268864/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759157.334709:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655268864/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759157.334714:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759157.334716:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759157.334717:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655268864:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759157.334720:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655268992:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759157.334729:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759157.334742:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759157/real 1787759157] req@ffff8c05a9f7aa00 x1874588655268992/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759157.334757:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655268992/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759157.334765:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759157.334768:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759157.334772:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655268992:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759157.462021:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759157.910089:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759158.485986:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759158.934123:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759159.509996:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759159.958032:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759160.533969:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759160.982102:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759161.557976:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759162.006043:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787759162.454067:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759162.454076:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787759162.454081:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759162.454090:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12218 00000100:00080000:2.0:1787759162.454100:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759162.454106:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759162.454142:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73800 x1874588655269120/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759162.454170:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759162.454176:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759162.454178:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759162.454182:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12218 00000100:00080000:2.0:1787759162.454186:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759162.454190:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759162.454200:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759162.454216:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588655269248/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759162.454226:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655269120:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787759162.454227:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759162.454249:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759162.454251:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759162.454255:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12218 00000100:00080000:2.0:1787759162.454258:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759162.454260:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759162.454264:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759162.454273:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588655269376/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759162.454281:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759162.454284:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759162.454285:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759162.454288:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12218 00000100:00080000:2.0:1787759162.454291:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759162.454293:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759162.454297:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759162.454304:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588655269504/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759162.454310:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787759162.454316:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787759162.454316:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759162.454321:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759162.454323:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12218 00000100:00080000:2.0:1787759162.454325:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759162.454328:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759162.454331:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759162.454338:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588655269632/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759162.454343:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759162.454346:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759162.454347:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759162.454349:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12218 00000100:00080000:2.0:1787759162.454351:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759162.454353:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759162.454356:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759162.454364:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588655269760/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759162.454369:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759162/real 1787759162] req@ffff8c05b2e73800 x1874588655269120/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759162.454371:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759162.454378:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787759162.454381:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759162.454388:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12218 00000100:00080000:2.0:1787759162.454392:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759162.454394:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759162.454397:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759162.454400:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73800 x1874588655269120/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759162.454406:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588655269888/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759162.454424:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759162.454425:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759162.454432:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759162.454433:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787759162.454436:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787759162.454441:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655269120:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787759162.454444:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12218 00000100:00080000:2.0:1787759162.454451:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759162.454456:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759162.454460:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759162.454471:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655269248:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759162.454473:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588655270016/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787759162.454502:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759162.454518:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759162/real 1787759162] req@ffff8c05b2e70e00 x1874588655269248/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759162.454530:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588655269248/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759162.454538:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759162.454541:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759162.454544:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655269248:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759162.454550:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655269376:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759162.454569:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759162.454579:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759162/real 1787759162] req@ffff8c05b2e71880 x1874588655269376/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759162.454591:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588655269376/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759162.454598:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759162.454601:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759162.454604:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655269376:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759162.454610:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655269504:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759162.454629:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759162.454639:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759162/real 1787759162] req@ffff8c05b2e71c00 x1874588655269504/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759162.454664:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588655269504/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759162.454672:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759162.454676:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759162.454679:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655269504:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759162.454684:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655269632:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759162.454704:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759162.454715:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759162/real 1787759162] req@ffff8c05b2e73b80 x1874588655269632/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759162.454732:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588655269632/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759162.454740:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759162.454743:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759162.454746:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655269632:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759162.454751:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655269760:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759162.454769:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759162.454780:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759162/real 1787759162] req@ffff8c05b2e71500 x1874588655269760/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759162.454791:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588655269760/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759162.454799:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759162.454802:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759162.454805:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655269760:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759162.454810:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655269888:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759162.454827:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759162.454837:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759162/real 1787759162] req@ffff8c05b2e73480 x1874588655269888/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759162.454847:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588655269888/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759162.454871:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759162.454919:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759162.454924:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655269888:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759162.454944:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655270016:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759162.454962:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759162.454972:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759162/real 1787759162] req@ffff8c05b2e70700 x1874588655270016/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759162.454997:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588655270016/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759162.455005:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759162.455008:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759162.455011:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655270016:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759162.581934:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759163.030431:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759163.605981:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759164.054033:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759164.630024:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759165.078056:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759165.654980:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759166.102074:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759166.677977:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759167.126126:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787759167.573983:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759167.573996:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787759167.573999:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759167.574008:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12223 00000100:00080000:2.0:1787759167.574017:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759167.574024:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759167.574068:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588655270144/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759167.574105:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759167.574109:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759167.574111:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759167.574115:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12223 00000100:00080000:2.0:1787759167.574118:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759167.574121:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759167.574128:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759167.574156:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588655270272/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759167.574164:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759167.574166:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759167.574168:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759167.574172:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12223 00000100:00080000:2.0:1787759167.574175:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759167.574180:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759167.574188:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655270144:192.168.201.107@tcp:250:kworker.0 00000008:00100000:2.0:1787759167.574190:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759167.574227:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588655270400/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759167.574237:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759167.574240:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759167.574242:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759167.574245:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12223 00000100:00080000:2.0:1787759167.574249:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759167.574252:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759167.574256:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759167.574264:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588655270528/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759167.574269:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759167.574271:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759167.574274:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787759167.574280:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787759167.574281:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12223 00000100:00080000:2.0:1787759167.574293:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759167.574297:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759167.574301:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759167.574312:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588655270656/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759167.574318:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759167.574320:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759167.574321:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759167.574324:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12223 00000100:00080000:2.0:1787759167.574326:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759167.574329:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759167.574334:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759167.574342:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759167/real 1787759167] req@ffff8c05b2e70380 x1874588655270144/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759167.574346:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588655270784/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759167.574402:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759167.574406:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787759167.574409:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588655270144/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759167.574409:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759167.574419:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12223 00000100:00080000:1.0:1787759167.574433:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759167.574433:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759167.574441:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:1.0:1787759167.574446:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000008:00100000:2.0:1787759167.574449:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759167.574455:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655270144:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787759167.574463:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588655270912/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759167.574470:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655270272:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787759167.574475:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759167.574478:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759167.574480:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759167.574484:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12223 00000100:00080000:2.0:1787759167.574486:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759167.574492:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759167.574495:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759167.574504:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588655271040/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787759167.574513:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759167.574528:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759167/real 1787759167] req@ffff8c05b2e72d80 x1874588655270272/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759167.574538:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588655270272/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759167.574543:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759167.574546:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759167.574548:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655270272:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759167.574573:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655270400:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759167.574592:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759167.574603:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759167/real 1787759167] req@ffff8c05b2e71180 x1874588655270400/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759167.574614:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588655270400/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759167.574621:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759167.574624:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759167.574627:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655270400:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759167.574633:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655270528:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759167.574650:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759167.574656:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759167/real 1787759167] req@ffff8c05b2e72680 x1874588655270528/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759167.574666:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588655270528/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759167.574673:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759167.574676:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759167.574679:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655270528:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759167.574685:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655270656:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759167.574702:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759167.574711:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759167/real 1787759167] req@ffff8c05b2e72a00 x1874588655270656/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759167.574718:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588655270656/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759167.574723:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759167.574725:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759167.574727:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655270656:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759167.574731:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655270784:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759167.574746:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759167.574756:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759167/real 1787759167] req@ffff8c05b2e72300 x1874588655270784/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759167.574772:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588655270784/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759167.574777:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759167.574779:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759167.574781:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655270784:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759167.574784:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655270912:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759167.574797:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759167.574802:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759167/real 1787759167] req@ffff8c05b2e70a80 x1874588655270912/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759167.574809:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588655270912/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759167.574813:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759167.574814:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759167.574816:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655270912:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759167.574819:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655271040:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759167.574828:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759167.574833:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759167/real 1787759167] req@ffff8c05b2e70000 x1874588655271040/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759167.574839:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588655271040/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759167.574847:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759167.574848:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759167.574850:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655271040:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759167.701979:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759168.150168:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759168.725978:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759169.174138:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759169.749996:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759170.198091:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759170.773953:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759171.222092:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759171.797989:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759172.246122:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787759172.694011:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759172.694021:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787759172.694026:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759172.694043:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12229 00000100:00080000:3.0:1787759172.694072:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759172.694081:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787759172.694125:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655271168/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759172.694160:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759172.694164:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759172.694167:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759172.694172:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12229 00000100:00080000:3.0:1787759172.694178:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759172.694182:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759172.694193:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759172.694207:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655271168:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787759172.694212:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655271296/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759172.694231:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759172.694235:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759172.694237:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759172.694242:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12229 00000100:00080000:3.0:1787759172.694246:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759172.694250:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759172.694255:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759172.694267:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655271424/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759172.694277:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787759172.694283:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759172.694333:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759172/real 1787759172] req@ffff8c05a9f7aa00 x1874588655271168/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759172.694350:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759172.694354:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759172.694361:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12229 00000100:00100000:2.0:1787759172.694366:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655271168/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759172.694379:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759172.694384:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759172.694388:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655271168:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787759172.694415:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655271296:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759172.694439:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759172.694454:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759172/real 1787759172] req@ffff8c05a9f7a680 x1874588655271296/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759172.694466:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655271296/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759172.694474:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759172.694478:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759172.694482:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655271296:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759172.694488:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655271424:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787759172.694489:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759172.694499:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759172.694509:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787759172.694520:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759172.694538:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759172/real 1787759172] req@ffff8c05a9f78a80 x1874588655271424/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759172.694563:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655271424/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759172.694566:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655271552/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759172.694614:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759172.694615:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759172.694627:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759172.694629:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787759172.694634:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787759172.694635:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655271424:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787759172.694649:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12229 00000100:00080000:3.0:1787759172.694655:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787759172.694661:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655271552:192.168.201.107@tcp:8:kworker.0 00000100:00080000:3.0:1787759172.694662:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759172.694673:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787759172.694685:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759172.694694:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655271680/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759172.694699:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759172/real 1787759172] req@ffff8c05a9f78380 x1874588655271552/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759172.694754:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655271552/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759172.694758:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759172.694765:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759172.694768:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759172.694768:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759172.694777:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787759172.694782:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12229 00000100:00100000:2.0:1787759172.694783:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655271552:192.168.201.107@tcp:8:kworker.0 00000100:00080000:3.0:1787759172.694799:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787759172.694811:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655271680:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787759172.694812:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759172.694826:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787759172.694834:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759172.694844:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655271808/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759172.694846:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759172/real 1787759172] req@ffff8c05a9f7b480 x1874588655271680/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759172.694914:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655271680/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759172.694922:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759172.694924:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759172.694933:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787759172.694933:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787759172.694941:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787759172.694943:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655271680:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787759172.694952:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12229 00000100:00080000:3.0:1787759172.694958:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787759172.694960:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655271808:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787759172.694968:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759172.694975:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787759172.694982:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759172.694996:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759172/real 1787759172] req@ffff8c05a9f78700 x1874588655271808/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759172.695000:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655271936/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759172.695030:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655271808/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759172.695039:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759172.695041:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759172.695061:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759172.695064:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787759172.695066:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787759172.695068:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655271808:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787759172.695080:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12229 00000100:00100000:2.0:1787759172.695085:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655271936:192.168.201.107@tcp:8:kworker.0 00000100:00080000:3.0:1787759172.695086:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759172.695096:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759172.695104:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787759172.695108:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759172.695124:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759172/real 1787759172] req@ffff8c05a9f79880 x1874588655271936/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759172.695133:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655272064/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759172.695139:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655271936/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759172.695175:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759172.695180:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759172.695184:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655271936:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759172.695198:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655272064:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759172.695214:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759172.695222:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759172/real 1787759172] req@ffff8c05a9f7a300 x1874588655272064/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759172.695229:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655272064/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759172.695233:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759172.695235:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759172.695237:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655272064:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759172.822011:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759173.270173:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759173.845974:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759174.294110:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759174.869975:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759175.318162:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759175.893984:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759176.342115:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759176.918055:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759177.366189:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787759177.814010:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759177.814020:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787759177.814024:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759177.814035:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12234 00000100:00080000:3.0:1787759177.814046:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759177.814053:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787759177.814091:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655272192/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759177.814126:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759177.814130:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759177.814132:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759177.814136:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12234 00000100:00080000:3.0:1787759177.814139:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759177.814143:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759177.814153:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759177.814166:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655272320/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759177.814178:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655272192:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787759177.814182:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759177.814187:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759177.814191:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759177.814203:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12234 00000100:00080000:3.0:1787759177.814207:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759177.814211:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759177.814217:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759177.814230:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655272448/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759177.814242:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759177.814246:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759177.814248:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759177.814251:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12234 00000100:00080000:3.0:1787759177.814257:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759177.814261:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759177.814266:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759177.814278:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655272576/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787759177.814285:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787759177.814298:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759177.814302:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759177.814304:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759177.814310:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12234 00000100:00080000:3.0:1787759177.814317:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759177.814321:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759177.814326:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759177.814338:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655272704/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759177.814347:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759177.814351:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759177.814352:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787759177.814359:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759177/real 1787759177] req@ffff8c05a9f79500 x1874588655272192/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759177.814371:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12234 00000100:00080000:3.0:1787759177.814379:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759177.814385:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759177.814390:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655272192/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787759177.814395:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787759177.814416:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787759177.814421:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655272832/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759177.814426:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759177.814439:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655272192:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787759177.814462:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759177.814465:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787759177.814468:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787759177.814476:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655272320:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787759177.814476:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12234 00000100:00080000:3.0:1787759177.814487:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759177.814495:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759177.814501:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759177.814514:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655272960/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787759177.814517:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787759177.814535:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787759177.814537:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759177/real 1787759177] req@ffff8c05a9f7b800 x1874588655272320/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759177.814542:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759177.814546:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759177.814563:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12234 00000100:00080000:3.0:1787759177.814571:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787759177.814576:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655272320/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759177.814582:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759177.814592:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787759177.814614:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787759177.814616:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655273088/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759177.814622:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759177.814638:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655272320:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759177.814658:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655272448:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759177.814684:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759177.814694:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759177/real 1787759177] req@ffff8c05a9f78e00 x1874588655272448/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759177.814705:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655272448/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759177.814713:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759177.814716:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759177.814719:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655272448:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759177.814725:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655272576:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759177.814740:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759177.814749:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759177/real 1787759177] req@ffff8c05a9f7ad80 x1874588655272576/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759177.814759:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655272576/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759177.814767:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759177.814770:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759177.814774:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655272576:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759177.814779:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655272704:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759177.814795:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759177.814803:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759177/real 1787759177] req@ffff8c05a9f7b100 x1874588655272704/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759177.814814:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655272704/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759177.814822:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759177.814825:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759177.814828:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655272704:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759177.814833:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655272832:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759177.814849:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759177.814857:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759177/real 1787759177] req@ffff8c05a9f79f80 x1874588655272832/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759177.814868:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655272832/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759177.814905:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759177.814909:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759177.814912:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655272832:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759177.814931:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655272960:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759177.814955:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759177.814980:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759177/real 1787759177] req@ffff8c05a9f7bb80 x1874588655272960/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759177.814991:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655272960/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759177.814999:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759177.815002:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759177.815005:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655272960:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759177.815010:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655273088:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759177.815026:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759177.815035:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759177/real 1787759177] req@ffff8c05a9f79180 x1874588655273088/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759177.815045:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655273088/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759177.815052:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759177.815055:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759177.815058:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655273088:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759177.942012:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759178.390081:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759178.965995:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759179.414172:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759179.989963:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759180.438106:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759181.013965:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759181.462169:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759182.037970:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759182.486010:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787759182.933993:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759182.934002:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787759182.934005:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759182.934013:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12239 00000100:00080000:3.0:1787759182.934022:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759182.934028:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787759182.934064:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655273216/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759182.934087:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759182.934089:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759182.934090:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759182.934094:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12239 00000100:00080000:3.0:1787759182.934096:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759182.934105:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759182.934113:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759182.934124:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655273344/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759182.934131:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759182.934133:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759182.934135:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787759182.934140:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655273216:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787759182.934142:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12239 00000100:00080000:3.0:1787759182.934148:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759182.934153:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759182.934159:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759182.934171:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655273472/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759182.934182:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759182.934185:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759182.934187:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759182.934191:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12239 00000100:00080000:3.0:1787759182.934203:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759182.934206:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759182.934210:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787759182.934214:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759182.934220:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655273600/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759182.934227:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759182.934230:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759182.934231:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759182.934234:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12239 00000100:00080000:3.0:1787759182.934237:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759182.934239:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759182.934242:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759182.934249:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655273728/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759182.934255:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759182.934257:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759182.934258:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759182.934261:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12239 00000100:00080000:3.0:1787759182.934263:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787759182.934266:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759182/real 1787759182] req@ffff8c05a9f79180 x1874588655273216/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759182.934268:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759182.934274:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759182.934284:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655273856/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759182.934291:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655273216/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759182.934292:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759182.934299:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787759182.934302:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759182.934308:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12239 00000100:00080000:2.0:1787759182.934310:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759182.934311:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759182.934317:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:2.0:1787759182.934319:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000008:00100000:3.0:1787759182.934321:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759182.934324:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655273216:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787759182.934330:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655273984/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759182.934336:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759182.934338:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759182.934339:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759182.934341:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12239 00000100:00080000:3.0:1787759182.934344:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759182.934346:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759182.934349:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759182.934356:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655273344:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759182.934356:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655274112/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787759182.934392:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759182.934405:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759182/real 1787759182] req@ffff8c05a9f7bb80 x1874588655273344/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759182.934413:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655273344/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759182.934419:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759182.934421:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759182.934423:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655273344:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759182.934427:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655273472:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759182.934439:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759182.934445:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759182/real 1787759182] req@ffff8c05a9f79f80 x1874588655273472/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759182.934451:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655273472/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759182.934457:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759182.934460:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759182.934463:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655273472:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759182.934469:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655273600:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759182.934497:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759182.934509:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759182/real 1787759182] req@ffff8c05a9f7b100 x1874588655273600/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759182.934520:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655273600/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759182.934528:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759182.934532:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759182.934534:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655273600:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759182.934538:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655273728:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759182.934552:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759182.934559:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759182/real 1787759182] req@ffff8c05a9f7ad80 x1874588655273728/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759182.934566:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655273728/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759182.934570:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759182.934572:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759182.934574:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655273728:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759182.934576:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655273856:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759182.934586:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759182.934591:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759182/real 1787759182] req@ffff8c05a9f78e00 x1874588655273856/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759182.934597:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655273856/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759182.934602:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759182.934603:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759182.934605:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655273856:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759182.934608:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655273984:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759182.934617:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759182.934622:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759182/real 1787759182] req@ffff8c05a9f7b800 x1874588655273984/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759182.934628:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655273984/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759182.934632:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759182.934633:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759182.934635:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655273984:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759182.934648:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655274112:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759182.934657:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759182.934662:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759182/real 1787759182] req@ffff8c05a9f79500 x1874588655274112/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759182.934668:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655274112/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759182.934676:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759182.934678:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759182.934679:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655274112:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759183.061975:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759183.510065:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759184.086011:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759184.534076:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759185.110031:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759185.558143:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759186.133987:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759186.582064:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759187.158122:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759187.606105:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787759188.054058:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759188.054069:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787759188.054072:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759188.054081:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12244 00000100:00080000:0.0:1787759188.054092:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759188.054098:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787759188.054133:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655274240/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759188.054155:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759188.054157:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787759188.054158:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759188.054162:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12244 00000100:00080000:0.0:1787759188.054164:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759188.054167:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759188.054175:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759188.054183:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655274368/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759188.054191:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759188.054193:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787759188.054194:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759188.054197:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12244 00000100:00080000:0.0:1787759188.054199:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759188.054204:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759188.054209:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759188.054220:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588655274496/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759188.054230:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759188.054236:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787759188.054243:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787759188.054245:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655274240:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787759188.054256:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12244 00000100:00080000:0.0:1787759188.054259:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759188.054263:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759188.054268:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759188.054279:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655274624/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759188.054286:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759188.054288:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787759188.054289:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759188.054291:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12244 00000100:00080000:0.0:1787759188.054294:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759188.054296:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759188.054298:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759188.054308:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588655274752/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759188.054314:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759188.054316:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787759188.054320:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759188.054322:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12244 00000100:00080000:0.0:1787759188.054324:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759188.054327:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759188.054331:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759188.054342:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655274880/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787759188.054348:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787759188.054361:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759188.054364:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787759188.054366:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759188.054373:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12244 00000100:00080000:0.0:1787759188.054378:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759188.054385:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759188.054390:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759188.054406:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759188/real 1787759188] req@ffff8c05a9f75880 x1874588655274240/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759188.054412:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655275008/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759188.054456:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759188.054459:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787759188.054461:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759188.054467:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12244 00000100:00100000:2.0:1787759188.054472:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655274240/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759188.054474:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759188.054488:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:2.0:1787759188.054491:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787759188.054496:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787759188.054497:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759188.054509:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655274240:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787759188.054517:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655275136/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759188.054523:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655274368:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759188.054564:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759188.054581:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759188/real 1787759188] req@ffff8c05a9f77800 x1874588655274368/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759188.054608:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655274368/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759188.054618:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759188.054621:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759188.054625:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655274368:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759188.054655:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655274496:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759188.054673:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759188.054681:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759188/real 1787759188] req@ffff8c05a9f75c00 x1874588655274496/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759188.054690:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588655274496/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759188.054696:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759188.054698:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759188.054701:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655274496:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759188.054705:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655274624:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759188.054717:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759188.054723:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759188/real 1787759188] req@ffff8c05a9f76300 x1874588655274624/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759188.054731:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655274624/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759188.054736:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759188.054738:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759188.054740:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655274624:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759188.054744:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655274752:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759188.054755:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759188.054761:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759188/real 1787759188] req@ffff8c05a9f76a00 x1874588655274752/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759188.054768:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588655274752/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759188.054773:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759188.054775:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759188.054777:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655274752:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759188.054794:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655274880:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759188.054809:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759188.054816:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759188/real 1787759188] req@ffff8c05a9f76680 x1874588655274880/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759188.054824:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655274880/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759188.054829:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759188.054831:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759188.054833:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655274880:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759188.054837:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655275008:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759188.054847:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759188.054853:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759188/real 1787759188] req@ffff8c05a9f74380 x1874588655275008/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759188.054860:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655275008/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759188.054870:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759188.054872:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759188.054874:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655275008:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759188.054907:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655275136:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759188.054920:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759188.054928:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759188/real 1787759188] req@ffff8c05a9f74a80 x1874588655275136/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759188.054935:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655275136/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759188.054940:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759188.054943:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759188.054944:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655275136:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759188.181998:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759188.630034:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759189.205978:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759189.654129:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759190.229958:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759190.678034:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759191.253980:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759191.702131:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759192.277970:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759192.726124:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787759193.174070:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759193.174080:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787759193.174083:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759193.174092:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12249 00000100:00080000:2.0:1787759193.174103:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759193.174110:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759193.174154:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588655275264/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759193.174185:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759193.174187:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759193.174189:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759193.174192:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12249 00000100:00080000:2.0:1787759193.174195:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759193.174197:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759193.174205:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759193.174218:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588655275392/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759193.174225:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759193.174227:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759193.174229:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759193.174231:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12249 00000100:00080000:2.0:1787759193.174233:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759193.174235:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759193.174237:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759193.174243:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655275264:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787759193.174246:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588655275520/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759193.174255:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759193.174256:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759193.174257:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759193.174260:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12249 00000100:00080000:2.0:1787759193.174261:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759193.174263:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759193.174267:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759193.174274:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588655275648/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759193.174281:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759193.174283:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759193.174285:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759193.174286:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12249 00000100:00080000:2.0:1787759193.174288:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759193.174291:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759193.174293:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759193.174299:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588655275776/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759193.174305:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787759193.174309:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787759193.174310:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759193.174313:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759193.174317:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12249 00000100:00080000:2.0:1787759193.174319:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759193.174322:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759193.174325:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759193.174332:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588655275904/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759193.174338:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759193.174340:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787759193.174342:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759193.174345:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12249 00000100:00080000:2.0:1787759193.174347:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759193.174350:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759193.174353:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759193.174358:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759193/real 1787759193] req@ffff8c05b2e73100 x1874588655275264/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759193.174361:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588655276032/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759193.174389:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759193.174392:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759193.174393:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759193.174396:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12249 00000100:00100000:1.0:1787759193.174399:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588655275264/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759193.174400:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759193.174407:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759193.174413:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787759193.174417:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787759193.174421:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588655276160/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759193.174422:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759193.174429:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655275264:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787759193.174454:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655275392:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759193.174473:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759193.174481:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759193/real 1787759193] req@ffff8c05b2e70000 x1874588655275392/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759193.174488:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588655275392/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759193.174493:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759193.174495:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759193.174497:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655275392:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759193.174501:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655275520:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759193.174511:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759193.174516:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759193/real 1787759193] req@ffff8c05b2e70a80 x1874588655275520/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759193.174522:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588655275520/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759193.174527:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759193.174529:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759193.174530:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655275520:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759193.174533:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655275648:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759193.174543:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759193.174548:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759193/real 1787759193] req@ffff8c05b2e72300 x1874588655275648/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759193.174555:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588655275648/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759193.174560:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759193.174561:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759193.174563:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655275648:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759193.174566:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655275776:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759193.174574:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759193.174580:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759193/real 1787759193] req@ffff8c05b2e72a00 x1874588655275776/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759193.174586:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588655275776/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759193.174591:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759193.174592:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759193.174594:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655275776:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759193.174596:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655275904:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759193.174605:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759193.174610:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759193/real 1787759193] req@ffff8c05b2e72680 x1874588655275904/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759193.174624:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588655275904/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759193.174633:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759193.174635:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759193.174637:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655275904:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759193.174640:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655276032:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759193.174651:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759193.174656:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759193/real 1787759193] req@ffff8c05b2e71180 x1874588655276032/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759193.174663:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588655276032/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759193.174667:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759193.174669:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759193.174670:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655276032:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759193.174673:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655276160:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759193.174684:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759193.174702:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759193/real 1787759193] req@ffff8c05b2e72d80 x1874588655276160/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759193.174714:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588655276160/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759193.174722:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759193.174725:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759193.174729:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655276160:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759193.302006:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759193.750164:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759194.326003:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759194.774108:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759195.349990:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759195.798265:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759196.373988:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759196.822038:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759197.397966:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759197.846035:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787759198.294031:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759198.294044:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787759198.294047:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759198.294058:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12254 00000100:00080000:0.0:1787759198.294070:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759198.294078:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787759198.294124:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588655276288/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759198.294159:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759198.294163:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787759198.294165:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759198.294170:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12254 00000100:00080000:0.0:1787759198.294174:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759198.294179:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759198.294190:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759198.294209:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655276416/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759198.294220:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759198.294227:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787759198.294230:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787759198.294230:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655276288:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787759198.294238:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12254 00000100:00080000:0.0:1787759198.294249:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759198.294255:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759198.294259:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759198.294273:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655276544/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759198.294282:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759198.294285:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787759198.294286:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759198.294290:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12254 00000100:00080000:0.0:1787759198.294317:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759198.294322:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759198.294328:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759198.294341:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655276672/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759198.294351:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759198.294354:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787759198.294357:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759198.294361:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12254 00000100:00080000:0.0:1787759198.294366:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759198.294372:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759198.294379:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759198.294404:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588655276800/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759198.294404:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655276416:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787759198.294431:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759198.294434:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787759198.294438:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759198.294444:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12254 00000100:00080000:0.0:1787759198.294451:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759198.294456:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759198.294462:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759198.294465:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655276544:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759198.294480:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655276928/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759198.294495:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759198.294499:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787759198.294503:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655276672:192.168.201.107@tcp:8:kworker.0 00000100:00080000:0.0:1787759198.294504:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759198.294517:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12254 00000100:00080000:0.0:1787759198.294525:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759198.294530:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759198.294535:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759198.294555:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588655277056/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759198.294564:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759198.294567:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787759198.294569:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787759198.294575:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655276800:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787759198.294580:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12254 00000100:00080000:0.0:1787759198.294589:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759198.294594:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759198.294611:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759198.294626:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655277184/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759198.294645:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655276928:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759198.294679:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655277056:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759198.294720:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655277184:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759198.421998:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759198.870074:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759199.446006:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759199.894166:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759200.469999:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759200.918096:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787759201.366200:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.107:988 00000400:00000100:2.0:1787759201.366263:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.107@tcp at host 192.168.201.107 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787759201.366289:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787759201.366339:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787759201.366366:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:1.0:1787759201.366389:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759198/real 1787759201] req@ffff8c05a9f74a80 x1874588655276416/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787759201.366412:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787759201.366447:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:1.0:1787759201.366450:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655276416/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759201.366476:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000800:00000100:2.0:1787759201.366489:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00080000:1.0:1787759201.366494:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759201.366507:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655276416:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759201.366526:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:1.0:1787759201.366529:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759198/real 1787759201] req@ffff8c05a9f74380 x1874588655276544/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787759201.366568:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:1.0:1787759201.366585:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655276544/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759201.366598:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759201.366603:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759201.366608:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655276544:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759201.366617:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759198/real 1787759201] req@ffff8c05a9f76680 x1874588655276672/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759201.366633:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655276672/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759201.366641:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759201.366646:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759201.366649:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655276672:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759201.366657:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759198/real 1787759201] req@ffff8c05a9f77b80 x1874588655276288/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759201.366671:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588655276288/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759201.366678:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759201.366682:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759201.366686:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655276288:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787759201.366725:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759198/real 1787759201] req@ffff8c05a9f77800 x1874588655277184/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759201.366739:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655277184/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759201.366748:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759201.366753:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759201.366757:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655277184:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759201.366766:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759198/real 1787759201] req@ffff8c05a9f76a00 x1874588655276800/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759201.366777:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588655276800/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759201.366785:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759201.366789:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759201.366793:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655276800:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759201.366802:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759198/real 1787759201] req@ffff8c05a9f76300 x1874588655276928/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759201.366813:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655276928/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759201.366821:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759201.366824:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759201.366827:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655276928:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759201.366835:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759198/real 1787759201] req@ffff8c05a9f75c00 x1874588655277056/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759201.366848:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588655277056/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759201.366856:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759201.366859:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759201.366862:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655277056:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759201.493974:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759201.942131:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759202.518002:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759202.966099:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787759203.414054:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759203.414064:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787759203.414070:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759203.414082:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12259 00000100:00080000:0.0:1787759203.414095:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759203.414103:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787759203.414142:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655277312/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759203.414179:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759203.414183:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787759203.414185:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759203.414190:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12259 00000100:00080000:0.0:1787759203.414194:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759203.414198:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759203.414210:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759203.414225:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588655277440/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759203.414237:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759203.414243:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787759203.414245:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787759203.414250:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655277312:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787759203.414251:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12259 00000100:00080000:0.0:1787759203.414259:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759203.414264:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759203.414269:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759203.414281:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588655277568/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759203.414299:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759203.414303:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787759203.414305:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759203.414310:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12259 00000100:00080000:0.0:1787759203.414329:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759203.414334:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759203.414340:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787759203.414344:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759203.414357:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588655277696/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759203.414369:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759203.414373:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787759203.414375:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759203.414378:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12259 00000100:00080000:0.0:1787759203.414382:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759203.414386:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759203.414392:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759203.414398:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759203/real 1787759203] req@ffff8c05a9f75880 x1874588655277312/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759203.414406:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588655277824/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759203.414425:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759203.414430:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787759203.414431:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655277312/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759203.414433:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759203.414441:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12259 00000100:00080000:0.0:1787759203.414444:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759203.414448:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759203.414453:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759203.414467:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588655277952/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759203.414467:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759203.414479:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787759203.414484:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787759203.414489:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655277312:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787759203.414490:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787759203.414494:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759203.414501:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12259 00000100:00080000:0.0:1787759203.414505:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759203.414531:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759203.414537:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759203.414542:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655277440:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759203.414553:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588655278080/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759203.414566:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759203.414570:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787759203.414572:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759203.414579:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12259 00000800:00000100:1.0:1787759203.414579:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787759203.414588:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759203.414593:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759203.414598:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759203.414601:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759203/real 1787759203] req@ffff8c05a9f76d80 x1874588655277440/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759203.414613:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655278208/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759203.414652:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588655277440/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759203.414665:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759203.414669:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759203.414673:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655277440:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759203.414682:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655277568:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759203.414706:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759203.414720:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759203/real 1787759203] req@ffff8c05a9f77100 x1874588655277568/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759203.414733:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588655277568/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759203.414742:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759203.414746:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759203.414749:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655277568:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759203.414755:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655277696:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759203.414778:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759203.414790:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759203/real 1787759203] req@ffff8c05a9f74e00 x1874588655277696/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759203.414804:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588655277696/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759203.414820:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759203.414823:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759203.414827:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655277696:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759203.414834:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655277824:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759203.414853:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759203.414864:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759203/real 1787759203] req@ffff8c05a9f75180 x1874588655277824/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759203.414907:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588655277824/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759203.414916:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759203.414920:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759203.414923:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655277824:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759203.414930:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655277952:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759203.414952:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759203.414970:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759203/real 1787759203] req@ffff8c05a9f74000 x1874588655277952/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759203.414982:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588655277952/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759203.414991:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759203.414994:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759203.414998:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655277952:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759203.415023:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655278080:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759203.415048:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759203.415065:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759203/real 1787759203] req@ffff8c05a9f75500 x1874588655278080/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759203.415077:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588655278080/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759203.415085:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759203.415088:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759203.415092:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655278080:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759203.415099:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655278208:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759203.415120:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759203.415131:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759203/real 1787759203] req@ffff8c05a9f75880 x1874588655278208/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759203.415143:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655278208/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759203.415152:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759203.415155:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759203.415158:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655278208:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759203.542006:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759203.990091:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759204.566029:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759205.014074:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759205.590150:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759206.038085:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759206.613998:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759207.062056:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759207.637981:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759208.086088:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787759208.533945:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759208.533954:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787759208.533959:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759208.533968:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12264 00000100:00080000:0.0:1787759208.533978:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759208.533984:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787759208.534018:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588655278336/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759208.534042:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759208.534046:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787759208.534048:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759208.534052:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12264 00000100:00080000:0.0:1787759208.534055:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759208.534058:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759208.534068:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759208.534081:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655278464/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759208.534083:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655278336:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787759208.534099:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759208.534102:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787759208.534103:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759208.534105:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12264 00000100:00080000:0.0:1787759208.534109:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759208.534112:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759208.534116:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759208.534126:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588655278592/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759208.534135:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759208.534138:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787759208.534140:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759208.534143:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12264 00000100:00080000:0.0:1787759208.534146:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759208.534149:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759208.534154:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787759208.534159:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759208.534168:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655278720/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759208.534176:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759208.534178:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787759208.534180:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759208.534183:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12264 00000100:00080000:0.0:1787759208.534186:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759208.534189:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759208.534192:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759208.534198:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759208/real 1787759208] req@ffff8c05a9f75c00 x1874588655278336/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759208.534203:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588655278848/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759208.534226:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759208.534229:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787759208.534233:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787759208.534233:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588655278336/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759208.534242:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12264 00000100:00080000:0.0:1787759208.534247:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759208.534253:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759208.534258:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787759208.534259:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759208.534267:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759208.534268:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655278976/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759208.534279:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655278336:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787759208.534281:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759208.534288:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787759208.534290:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759208.534294:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12264 00000100:00080000:0.0:1787759208.534296:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759208.534299:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759208.534302:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759208.534309:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655279104/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759208.534315:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655278464:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787759208.534320:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759208.534324:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787759208.534326:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759208.534329:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12264 00000100:00080000:0.0:1787759208.534333:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759208.534336:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759208.534340:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787759208.534344:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759208.534350:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655279232/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759208.534360:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759208/real 1787759208] req@ffff8c05a9f76300 x1874588655278464/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759208.534372:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655278464/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759208.534378:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759208.534382:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759208.534384:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655278464:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759208.534389:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655278592:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759208.534401:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759208.534415:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759208/real 1787759208] req@ffff8c05a9f76a00 x1874588655278592/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759208.534443:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588655278592/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759208.534448:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759208.534478:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759208.534481:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655278592:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759208.534486:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655278720:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759208.534503:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759208.534511:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759208/real 1787759208] req@ffff8c05a9f77800 x1874588655278720/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759208.534520:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655278720/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759208.534525:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759208.534527:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759208.534529:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655278720:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759208.534532:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655278848:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759208.534543:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759208.534549:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759208/real 1787759208] req@ffff8c05a9f77b80 x1874588655278848/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759208.534556:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588655278848/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759208.534587:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759208.534589:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759208.534592:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655278848:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759208.534596:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655278976:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759208.534609:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759208.534616:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759208/real 1787759208] req@ffff8c05a9f76680 x1874588655278976/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759208.534624:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655278976/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759208.534629:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759208.534631:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759208.534633:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655278976:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759208.534645:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655279104:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759208.534656:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759208.534662:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759208/real 1787759208] req@ffff8c05a9f74380 x1874588655279104/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759208.534669:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655279104/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759208.534674:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759208.534676:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759208.534678:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655279104:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759208.534682:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655279232:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759208.534691:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759208.534697:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759208/real 1787759208] req@ffff8c05a9f74a80 x1874588655279232/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759208.534703:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655279232/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759208.534708:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759208.534710:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759208.534712:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655279232:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759208.661988:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759209.110077:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759209.685946:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759210.134065:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759210.710105:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759211.158032:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759211.733994:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759212.182102:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759212.757990:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759213.205991:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787759213.653980:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759213.653988:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787759213.653992:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759213.654001:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12270 00000100:00080000:1.0:1787759213.654011:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759213.654017:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759213.654052:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655279360/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759213.654078:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759213.654080:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759213.654081:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759213.654084:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12270 00000100:00080000:1.0:1787759213.654086:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759213.654088:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759213.654097:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759213.654109:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655279488/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759213.654118:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759213.654122:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759213.654123:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759213.654128:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12270 00000100:00080000:1.0:1787759213.654131:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759213.654136:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759213.654138:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655279360:192.168.201.107@tcp:250:kworker.0 00000008:00100000:1.0:1787759213.654142:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759213.654155:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655279616/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759213.654162:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759213.654164:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759213.654166:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759213.654168:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12270 00000100:00080000:1.0:1787759213.654170:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759213.654173:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759213.654177:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759213.654184:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655279744/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759213.654190:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759213.654192:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759213.654192:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759213.654194:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12270 00000100:00080000:1.0:1787759213.654198:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759213.654200:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759213.654203:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787759213.654208:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759213.654211:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655279872/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759213.654218:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759213.654220:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759213.654222:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759213.654225:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12270 00000100:00080000:1.0:1787759213.654233:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759213.654235:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759213.654238:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759213.654245:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655280000/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759213.654250:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759213.654254:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787759213.654254:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759213/real 1787759213] req@ffff8c05b2d9b480 x1874588655279360/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759213.654257:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759213.654264:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12270 00000100:00080000:1.0:1787759213.654268:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759213.654271:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759213.654275:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759213.654279:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655279360/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759213.654283:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655280128/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759213.654297:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759213.654297:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759213.654303:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759213.654305:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759213.654305:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787759213.654312:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12270 00000100:00100000:2.0:1787759213.654313:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655279360:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787759213.654315:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759213.654322:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759213.654325:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655279488:192.168.201.107@tcp:38:kworker.0 00000008:00100000:1.0:1787759213.654327:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759213.654348:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655280256/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787759213.654363:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759213.654374:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759213/real 1787759213] req@ffff8c05b2d9aa00 x1874588655279488/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759213.654382:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655279488/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759213.654388:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759213.654390:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759213.654392:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655279488:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759213.654412:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655279616:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759213.654423:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759213.654430:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759213/real 1787759213] req@ffff8c05b2d99f80 x1874588655279616/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759213.654437:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655279616/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759213.654441:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759213.654443:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759213.654445:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655279616:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759213.654448:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655279744:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759213.654457:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759213.654462:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759213/real 1787759213] req@ffff8c05b2d9bb80 x1874588655279744/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759213.654469:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655279744/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759213.654474:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759213.654475:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759213.654477:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655279744:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759213.654479:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655279872:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759213.654489:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759213.654494:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759213/real 1787759213] req@ffff8c05b2d9b100 x1874588655279872/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759213.654501:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655279872/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759213.654506:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759213.654507:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759213.654509:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655279872:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759213.654512:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655280000:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759213.654520:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759213.654525:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759213/real 1787759213] req@ffff8c05b2d98000 x1874588655280000/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759213.654531:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655280000/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759213.654536:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759213.654538:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759213.654539:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655280000:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759213.654542:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655280128:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759213.654550:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759213.654555:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759213/real 1787759213] req@ffff8c05b2d98700 x1874588655280128/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759213.654561:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655280128/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759213.654565:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759213.654567:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759213.654569:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655280128:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759213.654572:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655280256:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759213.654580:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759213.654585:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759213/real 1787759213] req@ffff8c05b2d99880 x1874588655280256/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759213.654591:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655280256/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759213.654595:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759213.654597:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759213.654598:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655280256:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759213.781980:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759214.230105:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759214.805939:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759215.254001:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759215.829950:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759216.278105:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759216.853941:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759217.302103:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759217.877978:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759218.326103:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787759218.774039:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759218.774048:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787759218.774052:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759218.774061:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12275 00000100:00080000:2.0:1787759218.774071:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759218.774077:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759218.774114:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588655280384/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759218.774141:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759218.774144:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759218.774145:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759218.774147:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12275 00000100:00080000:2.0:1787759218.774150:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759218.774153:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759218.774163:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759218.774178:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588655280512/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759218.774190:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759218.774198:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787759218.774200:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655280384:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787759218.774202:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759218.774214:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12275 00000100:00080000:2.0:1787759218.774218:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759218.774220:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759218.774224:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759218.774236:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588655280640/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759218.774260:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759218.774263:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759218.774264:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759218.774267:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12275 00000100:00080000:2.0:1787759218.774270:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759218.774274:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759218.774277:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759218.774285:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588655280768/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759218.774291:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759218.774294:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759218.774296:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759218.774298:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12275 00000100:00080000:2.0:1787759218.774301:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787759218.774308:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787759218.774309:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759218.774316:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759218.774325:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588655280896/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759218.774332:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759218.774334:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759218.774335:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759218.774338:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12275 00000100:00080000:2.0:1787759218.774340:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759218.774343:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759218.774346:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759218.774356:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588655281024/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759218.774368:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759218/real 1787759218] req@ffff8c05b2e70380 x1874588655280384/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759218.774368:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759218.774386:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787759218.774392:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787759218.774400:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588655280384/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759218.774401:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12275 00000100:00080000:2.0:1787759218.774411:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759218.774423:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759218.774441:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787759218.774444:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759218.774449:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759218.774457:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655280384:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787759218.774458:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588655281152/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759218.774479:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759218.774482:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759218.774484:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759218.774489:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12275 00000100:00080000:2.0:1787759218.774491:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759218.774495:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759218.774499:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655280512:192.168.201.107@tcp:38:kworker.0 00000008:00100000:2.0:1787759218.774502:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759218.774518:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588655281280/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787759218.774531:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759218.774552:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759218/real 1787759218] req@ffff8c05b2e70700 x1874588655280512/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759218.774567:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588655280512/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759218.774577:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759218.774581:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759218.774585:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655280512:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759218.774590:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655280640:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759218.774609:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759218.774616:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759218/real 1787759218] req@ffff8c05b2e73480 x1874588655280640/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759218.774624:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588655280640/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759218.774630:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759218.774632:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759218.774634:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655280640:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759218.774639:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655280768:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759218.774651:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759218.774657:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759218/real 1787759218] req@ffff8c05b2e71500 x1874588655280768/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759218.774664:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588655280768/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759218.774670:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759218.774672:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759218.774673:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655280768:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759218.774677:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655280896:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759218.774687:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759218.774693:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759218/real 1787759218] req@ffff8c05b2e73b80 x1874588655280896/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759218.774700:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588655280896/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759218.774705:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759218.774707:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759218.774709:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655280896:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759218.774712:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655281024:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759218.774722:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759218.774728:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759218/real 1787759218] req@ffff8c05b2e71c00 x1874588655281024/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759218.774735:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588655281024/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759218.774741:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759218.774743:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759218.774745:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655281024:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759218.774748:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655281152:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759218.774757:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759218.774763:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759218/real 1787759218] req@ffff8c05b2e71880 x1874588655281152/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759218.774770:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588655281152/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759218.774775:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759218.774777:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759218.774779:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655281152:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759218.774792:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655281280:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759218.774802:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759218.774807:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759218/real 1787759218] req@ffff8c05b2e70e00 x1874588655281280/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759218.774814:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588655281280/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759218.774824:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759218.774826:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759218.774828:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655281280:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759218.901951:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759219.350028:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759219.926000:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759220.374152:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759220.950023:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759221.398089:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759221.974010:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759222.422168:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759222.998036:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759223.446128:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787759223.894033:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759223.894044:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787759223.894048:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759223.894057:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12280 00000100:00080000:2.0:1787759223.894069:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759223.894075:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759223.894115:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73800 x1874588655281408/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759223.894144:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759223.894146:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759223.894148:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759223.894165:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12280 00000100:00080000:2.0:1787759223.894169:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759223.894172:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759223.894180:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759223.894208:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588655281536/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759223.894221:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787759223.894223:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655281408:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787759223.894227:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759223.894231:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759223.894238:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12280 00000100:00080000:2.0:1787759223.894241:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759223.894244:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759223.894248:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759223.894259:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588655281664/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759223.894266:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759223.894275:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759223.894276:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759223.894279:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12280 00000100:00080000:2.0:1787759223.894282:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759223.894285:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759223.894288:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759223.894296:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588655281792/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787759223.894301:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787759223.894312:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759223.894314:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759223.894315:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759223.894319:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12280 00000100:00080000:2.0:1787759223.894322:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759223.894325:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759223.894328:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759223.894335:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588655281920/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759223.894342:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759223.894344:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759223.894345:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759223.894349:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12280 00000100:00080000:2.0:1787759223.894351:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759223.894354:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759223.894358:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759223.894359:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759223/real 1787759223] req@ffff8c05b2e73800 x1874588655281408/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759223.894368:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588655282048/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759223.894380:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759223.894382:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787759223.894384:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787759223.894389:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73800 x1874588655281408/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759223.894389:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12280 00000100:00080000:2.0:1787759223.894393:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759223.894399:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759223.894405:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787759223.894416:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787759223.894417:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588655282176/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759223.894427:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787759223.894429:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759223.894434:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759223.894437:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787759223.894438:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655281408:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787759223.894440:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12280 00000100:00080000:2.0:1787759223.894442:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759223.894444:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759223.894447:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759223.894453:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588655282304/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759223.894483:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655281536:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759223.894509:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759223.894522:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759223/real 1787759223] req@ffff8c05b2e70e00 x1874588655281536/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759223.894535:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588655281536/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759223.894552:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759223.894555:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759223.894559:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655281536:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759223.894566:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655281664:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759223.894594:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759223.894605:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759223/real 1787759223] req@ffff8c05b2e71880 x1874588655281664/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759223.894618:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588655281664/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759223.894626:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759223.894630:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759223.894634:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655281664:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759223.894640:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655281792:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759223.894657:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759223.894668:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759223/real 1787759223] req@ffff8c05b2e71c00 x1874588655281792/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759223.894681:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588655281792/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759223.894690:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759223.894701:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759223.894706:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655281792:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759223.894712:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655281920:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759223.894729:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759223.894740:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759223/real 1787759223] req@ffff8c05b2e73b80 x1874588655281920/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759223.894752:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588655281920/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759223.894768:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759223.894772:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759223.894775:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655281920:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759223.894782:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655282048:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759223.894802:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759223.894814:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759223/real 1787759223] req@ffff8c05b2e71500 x1874588655282048/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759223.894827:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588655282048/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759223.894837:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759223.894840:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759223.894844:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655282048:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759223.894850:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655282176:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759223.894866:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759223.894902:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759223/real 1787759223] req@ffff8c05b2e73480 x1874588655282176/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759223.894916:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588655282176/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759223.894926:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759223.894930:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759223.894933:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655282176:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759223.894939:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655282304:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759223.894958:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759223.894968:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759223/real 1787759223] req@ffff8c05b2e70700 x1874588655282304/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759223.894981:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588655282304/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759223.894990:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759223.894994:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759223.894997:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655282304:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759224.022003:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759224.470175:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759225.045976:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759225.494019:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759226.070004:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759226.518209:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759227.093997:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759227.542040:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759228.117994:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759228.566091:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787759229.014040:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759229.014058:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787759229.014072:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759229.014084:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12285 00000100:00080000:0.0:1787759229.014102:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759229.014113:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787759229.014169:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655282432/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759229.014200:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759229.014204:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787759229.014205:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759229.014210:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12285 00000100:00080000:0.0:1787759229.014214:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759229.014218:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759229.014228:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759229.014243:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655282560/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759229.014256:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655282432:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787759229.014262:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759229.014265:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787759229.014266:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759229.014270:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12285 00000100:00080000:0.0:1787759229.014272:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759229.014282:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759229.014287:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759229.014297:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655282688/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759229.014304:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759229.014306:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787759229.014308:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759229.014310:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12285 00000100:00080000:0.0:1787759229.014315:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759229.014318:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759229.014321:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787759229.014329:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759229.014330:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588655282816/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759229.014337:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759229.014340:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787759229.014341:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759229.014343:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12285 00000100:00080000:0.0:1787759229.014346:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759229.014349:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759229.014352:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759229.014358:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655282944/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759229.014364:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759229.014366:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787759229.014368:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759229.014371:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12285 00000100:00080000:0.0:1787759229.014375:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759229.014379:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759229.014383:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759229/real 1787759229] req@ffff8c05a9f74a80 x1874588655282432/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787759229.014385:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759229.014402:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588655283072/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759229.014413:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759229.014417:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787759229.014420:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787759229.014420:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655282432/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759229.014428:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12285 00000100:00080000:0.0:1787759229.014434:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759229.014439:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759229.014444:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787759229.014444:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759229.014453:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759229.014457:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655283200/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759229.014461:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655282432:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787759229.014474:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759229.014476:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787759229.014478:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759229.014480:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12285 00000100:00080000:0.0:1787759229.014482:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759229.014484:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759229.014486:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759229.014501:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655282560:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759229.014502:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588655283328/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787759229.014530:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759229.014542:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759229/real 1787759229] req@ffff8c05a9f74380 x1874588655282560/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759229.014551:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655282560/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759229.014557:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759229.014559:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759229.014561:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655282560:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759229.014565:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655282688:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759229.014576:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759229.014582:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759229/real 1787759229] req@ffff8c05a9f76680 x1874588655282688/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759229.014589:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655282688/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759229.014597:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759229.014599:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759229.014601:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655282688:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759229.014604:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655282816:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759229.014614:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759229.014618:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759229/real 1787759229] req@ffff8c05a9f77b80 x1874588655282816/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759229.014625:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588655282816/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759229.014630:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759229.014631:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759229.014633:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655282816:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759229.014636:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655282944:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759229.014645:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759229.014650:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759229/real 1787759229] req@ffff8c05a9f77800 x1874588655282944/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759229.014656:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655282944/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759229.014664:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759229.014667:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759229.014670:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655282944:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759229.014676:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655283072:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759229.014704:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759229.014715:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759229/real 1787759229] req@ffff8c05a9f76a00 x1874588655283072/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759229.014727:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588655283072/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759229.014734:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759229.014737:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759229.014741:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655283072:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759229.014746:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655283200:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759229.014771:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759229.014782:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759229/real 1787759229] req@ffff8c05a9f76300 x1874588655283200/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759229.014794:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655283200/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759229.014802:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759229.014805:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759229.014808:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655283200:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759229.014826:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655283328:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759229.014843:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759229.014853:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759229/real 1787759229] req@ffff8c05a9f75c00 x1874588655283328/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759229.014861:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588655283328/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759229.014866:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759229.014868:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759229.014870:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655283328:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759229.141959:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759229.590121:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759230.165998:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759230.614137:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759231.190042:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759231.639069:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759232.213956:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759232.662179:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759233.238046:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759233.686121:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787759234.134061:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759234.134074:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787759234.134078:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759234.134090:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12290 00000100:00080000:2.0:1787759234.134102:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759234.134110:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759234.134151:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588655283456/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759234.134175:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759234.134179:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759234.134180:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759234.134185:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12290 00000100:00080000:2.0:1787759234.134189:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759234.134193:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759234.134202:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759234.134217:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588655283584/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759234.134229:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787759234.134234:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655283456:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787759234.134234:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759234.134245:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759234.134251:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12290 00000100:00080000:2.0:1787759234.134255:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759234.134259:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759234.134265:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759234.134278:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588655283712/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759234.134288:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759234.134292:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759234.134294:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787759234.134299:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787759234.134302:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12290 00000100:00080000:2.0:1787759234.134308:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759234.134313:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759234.134319:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759234.134331:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588655283840/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759234.134340:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759234/real 1787759234] req@ffff8c05b2e70380 x1874588655283456/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759234.134344:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759234.134351:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759234.134354:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759234.134360:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12290 00000100:00080000:2.0:1787759234.134364:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787759234.134369:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588655283456/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759234.134371:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759234.134378:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759234.134388:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588655283968/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759234.134400:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759234.134401:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759234.134407:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759234.134409:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787759234.134410:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759234.134418:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12290 00000100:00100000:1.0:1787759234.134420:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655283456:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787759234.134421:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759234.134428:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759234.134433:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759234.134444:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588655284096/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759234.134452:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655283584:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787759234.134456:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759234.134460:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787759234.134463:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759234.134469:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12290 00000100:00080000:2.0:1787759234.134473:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759234.134477:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759234.134482:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787759234.134493:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759234.134496:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588655284224/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759234.134507:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787759234.134511:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759234/real 1787759234] req@ffff8c05b2e72d80 x1874588655283584/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759234.134512:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759234.134515:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759234.134531:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12290 00000100:00080000:2.0:1787759234.134537:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787759234.134539:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588655283584/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759234.134544:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759234.134553:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787759234.134556:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759234.134562:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759234.134568:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588655284352/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759234.134569:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655283584:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759234.134582:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655283712:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759234.134606:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759234.134621:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759234/real 1787759234] req@ffff8c05b2e71180 x1874588655283712/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759234.134633:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588655283712/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759234.134641:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759234.134645:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759234.134648:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655283712:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759234.134654:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655283840:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759234.134673:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759234.134683:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759234/real 1787759234] req@ffff8c05b2e72680 x1874588655283840/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759234.134694:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588655283840/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759234.134707:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759234.134710:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759234.134714:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655283840:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759234.134719:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655283968:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759234.134739:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759234.134761:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759234/real 1787759234] req@ffff8c05b2e72a00 x1874588655283968/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759234.134772:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588655283968/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759234.134781:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759234.134784:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759234.134787:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655283968:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759234.134803:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655284096:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759234.134823:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759234.134834:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759234/real 1787759234] req@ffff8c05b2e72300 x1874588655284096/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759234.134845:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588655284096/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759234.134853:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759234.134856:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759234.134860:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655284096:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759234.134908:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655284224:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759234.134928:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759234.134939:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759234/real 1787759234] req@ffff8c05b2e70a80 x1874588655284224/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759234.134950:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588655284224/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759234.134958:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759234.134961:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759234.134981:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655284224:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759234.134989:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655284352:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759234.135007:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759234.135017:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759234/real 1787759234] req@ffff8c05b2e70000 x1874588655284352/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759234.135028:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588655284352/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759234.135036:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759234.135039:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759234.135042:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655284352:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759234.262018:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759234.710169:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759235.286424:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759235.734120:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759236.309989:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759236.758112:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759237.334163:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759237.782047:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759238.357975:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759238.806152:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787759239.253974:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759239.253984:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787759239.253987:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759239.253995:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12295 00000100:00080000:2.0:1787759239.254006:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759239.254012:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759239.254047:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588655284480/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759239.254074:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759239.254077:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759239.254078:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759239.254081:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12295 00000100:00080000:2.0:1787759239.254084:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759239.254087:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759239.254095:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759239.254105:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655284480:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787759239.254110:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588655284608/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759239.254122:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759239.254126:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759239.254127:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759239.254132:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12295 00000100:00080000:2.0:1787759239.254136:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759239.254141:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759239.254146:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759239.254157:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588655284736/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787759239.254171:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787759239.254173:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759239.254176:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759239.254178:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759239.254181:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12295 00000100:00080000:2.0:1787759239.254186:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759239.254189:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759239.254195:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759239.254204:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588655284864/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759239.254210:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759239.254212:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759239.254214:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787759239.254218:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759239/real 1787759239] req@ffff8c05b2e73100 x1874588655284480/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759239.254219:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12295 00000100:00080000:2.0:1787759239.254224:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759239.254230:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759239.254235:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759239.254246:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588655284992/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759239.254250:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588655284480/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759239.254255:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759239.254262:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759239.254264:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759239.254270:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12295 00000100:00080000:1.0:1787759239.254273:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759239.254274:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759239.254279:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:1.0:1787759239.254281:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000008:00100000:2.0:1787759239.254283:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759239.254286:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655284480:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787759239.254294:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588655285120/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759239.254304:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759239.254307:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787759239.254310:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655284608:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787759239.254310:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759239.254316:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12295 00000100:00080000:2.0:1787759239.254319:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759239.254321:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759239.254324:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759239.254332:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588655285248/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787759239.254337:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787759239.254345:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759239.254348:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759239.254349:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787759239.254352:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759239/real 1787759239] req@ffff8c05b2e70000 x1874588655284608/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759239.254353:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12295 00000100:00080000:2.0:1787759239.254358:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759239.254364:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759239.254369:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759239.254372:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588655284608/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759239.254379:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588655285376/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759239.254380:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759239.254387:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759239.254393:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655284608:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759239.254400:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655284736:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759239.254413:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759239.254419:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759239/real 1787759239] req@ffff8c05b2e70a80 x1874588655284736/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759239.254427:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588655284736/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759239.254431:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759239.254433:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759239.254435:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655284736:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759239.254438:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655284864:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759239.254447:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759239.254452:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759239/real 1787759239] req@ffff8c05b2e72300 x1874588655284864/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759239.254459:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588655284864/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759239.254463:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759239.254464:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759239.254466:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655284864:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759239.254469:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655284992:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759239.254477:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759239.254482:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759239/real 1787759239] req@ffff8c05b2e72a00 x1874588655284992/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759239.254489:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588655284992/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759239.254497:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759239.254499:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759239.254501:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655284992:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759239.254504:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655285120:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759239.254512:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759239.254516:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759239/real 1787759239] req@ffff8c05b2e72680 x1874588655285120/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759239.254522:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588655285120/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759239.254526:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759239.254528:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759239.254537:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655285120:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759239.254549:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655285248:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759239.254559:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759239.254565:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759239/real 1787759239] req@ffff8c05b2e71180 x1874588655285248/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759239.254572:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588655285248/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759239.254577:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759239.254579:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759239.254581:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655285248:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759239.254584:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655285376:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759239.254593:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759239.254598:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759239/real 1787759239] req@ffff8c05b2e72d80 x1874588655285376/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759239.254605:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588655285376/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759239.254609:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759239.254611:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759239.254612:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655285376:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759239.382003:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759239.830008:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759240.406024:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759240.854083:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759241.430004:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759241.878509:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759242.453994:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759242.902136:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759243.477930:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759243.926094:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787759244.374036:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759244.374049:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787759244.374052:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759244.374062:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12300 00000100:00080000:0.0:1787759244.374072:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759244.374079:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787759244.374117:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655285504/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759244.374145:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759244.374148:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787759244.374150:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759244.374154:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12300 00000100:00080000:0.0:1787759244.374157:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759244.374160:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759244.374168:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759244.374181:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588655285632/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759244.374183:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655285504:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787759244.374197:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759244.374200:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787759244.374201:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759244.374205:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12300 00000100:00080000:0.0:1787759244.374216:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759244.374220:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759244.374225:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759244.374240:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588655285760/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759244.374249:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759244.374252:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787759244.374253:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759244.374256:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12300 00000100:00080000:0.0:1787759244.374262:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759244.374265:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759244.374269:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787759244.374275:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759244.374282:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588655285888/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759244.374290:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759244.374292:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787759244.374294:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759244.374296:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12300 00000100:00080000:0.0:1787759244.374299:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759244.374301:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759244.374305:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759244.374313:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588655286016/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759244.374319:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759244.374322:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787759244.374322:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759244/real 1787759244] req@ffff8c05a9f75880 x1874588655285504/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759244.374326:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759244.374332:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12300 00000100:00080000:0.0:1787759244.374337:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759244.374343:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759244.374351:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759244.374364:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655285504/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759244.374367:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588655286144/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759244.374383:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759244.374385:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759244.374390:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787759244.374391:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787759244.374396:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787759244.374396:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655285504:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787759244.374406:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12300 00000100:00080000:0.0:1787759244.374411:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759244.374415:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759244.374422:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759244.374423:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655285632:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759244.374437:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588655286272/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759244.374446:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787759244.374448:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787759244.374450:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787759244.374454:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759244.374460:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12300 00000100:00100000:2.0:1787759244.374461:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759244/real 1787759244] req@ffff8c05a9f75500 x1874588655285632/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759244.374467:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759244.374476:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759244.374476:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588655285632/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787759244.374484:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787759244.374486:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759244.374490:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759244.374492:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655285632:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759244.374497:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655285760:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759244.374500:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655286400/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787759244.374514:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759244.374524:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759244/real 1787759244] req@ffff8c05a9f74000 x1874588655285760/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759244.374538:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588655285760/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759244.374552:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759244.374555:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759244.374559:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655285760:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759244.374565:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655285888:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759244.374585:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759244.374594:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759244/real 1787759244] req@ffff8c05a9f75180 x1874588655285888/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759244.374601:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588655285888/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759244.374607:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759244.374609:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759244.374611:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655285888:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759244.374616:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655286016:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759244.374627:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759244.374633:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759244/real 1787759244] req@ffff8c05a9f74e00 x1874588655286016/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759244.374641:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588655286016/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759244.374647:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759244.374649:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759244.374652:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655286016:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759244.374655:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655286144:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759244.374669:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759244.374676:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759244/real 1787759244] req@ffff8c05a9f77100 x1874588655286144/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759244.374688:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588655286144/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759244.374696:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759244.374699:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759244.374702:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655286144:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759244.374729:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655286272:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759244.374745:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759244.374753:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759244/real 1787759244] req@ffff8c05a9f76d80 x1874588655286272/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759244.374760:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588655286272/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759244.374766:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759244.374768:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759244.374771:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655286272:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759244.374775:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655286400:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759244.374785:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759244.374806:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759244/real 1787759244] req@ffff8c05a9f75880 x1874588655286400/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759244.374814:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655286400/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759244.374824:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759244.374826:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759244.374828:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655286400:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759244.501969:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759244.950081:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759245.526001:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759245.974094:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759246.549976:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759246.998093:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759247.573983:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759248.022148:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759248.597946:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759249.046080:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787759249.493969:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759249.493982:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787759249.493986:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759249.494005:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12305 00000100:00080000:2.0:1787759249.494018:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759249.494028:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759249.494069:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588655286528/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759249.494103:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759249.494107:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759249.494109:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759249.494115:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12305 00000100:00080000:2.0:1787759249.494119:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759249.494124:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759249.494135:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759249.494162:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588655286656/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759249.494177:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759249.494182:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759249.494185:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787759249.494190:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655286528:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787759249.494194:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12305 00000100:00080000:2.0:1787759249.494207:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759249.494215:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759249.494222:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759249.494241:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588655286784/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759249.494254:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759249.494258:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759249.494260:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759249.494264:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12305 00000100:00080000:2.0:1787759249.494269:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759249.494274:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759249.494280:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787759249.494290:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759249.494302:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588655286912/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759249.494317:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759249.494321:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759249.494323:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759249.494330:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12305 00000100:00080000:2.0:1787759249.494337:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787759249.494341:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759249/real 1787759249] req@ffff8c05b2e70380 x1874588655286528/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759249.494349:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759249.494360:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759249.494376:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588655286528/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759249.494383:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588655287040/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759249.494412:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759249.494415:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759249.494422:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759249.494424:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787759249.494427:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787759249.494434:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655286528:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787759249.494439:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12305 00000100:00080000:2.0:1787759249.494447:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759249.494453:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759249.494460:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759249.494472:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655286656:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759249.494490:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588655287168/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759249.494506:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787759249.494512:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787759249.494515:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787759249.494519:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759249.494526:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12305 00000100:00080000:2.0:1787759249.494531:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787759249.494538:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759249/real 1787759249] req@ffff8c05b2e70700 x1874588655286656/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759249.494539:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759249.494550:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759249.494560:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588655286656/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759249.494569:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787759249.494571:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588655287296/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759249.494576:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759249.494583:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655286656:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759249.494604:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655286784:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787759249.494624:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759249.494629:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1787759249.494633:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787759249.494635:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759249.494645:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12305 00000100:00100000:1.0:1787759249.494648:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759249/real 1787759249] req@ffff8c05b2e73480 x1874588655286784/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759249.494659:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588655286784/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759249.494663:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759249.494669:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759249.494675:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:1.0:1787759249.494678:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759249.494689:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655286784:192.168.201.107@tcp:38:kworker.0 00000008:00100000:2.0:1787759249.494691:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759249.494726:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588655287424/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759249.494727:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655286912:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759249.494778:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759249.494793:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759249/real 1787759249] req@ffff8c05b2e71500 x1874588655286912/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759249.494806:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588655286912/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759249.494813:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759249.494815:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759249.494817:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655286912:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759249.494822:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655287040:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759249.494838:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759249.494845:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759249/real 1787759249] req@ffff8c05b2e73b80 x1874588655287040/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759249.494853:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588655287040/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759249.494858:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759249.494860:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759249.494862:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655287040:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759249.494913:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655287168:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759249.494927:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759249.494934:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759249/real 1787759249] req@ffff8c05b2e71c00 x1874588655287168/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759249.494942:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588655287168/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759249.494952:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759249.494954:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759249.494956:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655287168:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759249.494960:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655287296:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759249.494971:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759249.494977:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759249/real 1787759249] req@ffff8c05b2e71880 x1874588655287296/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759249.494984:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588655287296/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759249.494989:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759249.494991:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759249.494992:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655287296:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759249.494996:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655287424:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759249.495006:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759249.495012:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759249/real 1787759249] req@ffff8c05b2e70e00 x1874588655287424/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759249.495019:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588655287424/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759249.495024:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759249.495025:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759249.495028:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655287424:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759249.621995:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759250.070113:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759250.646008:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759251.094105:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759251.670009:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759252.118118:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759252.694009:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759253.142156:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759253.717999:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759254.166072:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787759254.614017:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759254.614038:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787759254.614043:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759254.614053:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12311 00000100:00080000:1.0:1787759254.614063:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759254.614071:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759254.614110:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588655287552/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759254.614136:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759254.614139:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759254.614140:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759254.614144:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12311 00000100:00080000:1.0:1787759254.614147:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759254.614149:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759254.614163:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759254.614173:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655287680/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759254.614183:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655287552:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787759254.614186:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759254.614196:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759254.614198:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759254.614204:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12311 00000100:00080000:1.0:1787759254.614208:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759254.614213:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759254.614219:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759254.614236:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655287808/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759254.614246:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759254.614249:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759254.614251:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759254.614255:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12311 00000100:00080000:1.0:1787759254.614258:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759254.614261:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759254.614266:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787759254.614267:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759254.614279:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655287936/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759254.614287:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759254.614290:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759254.614291:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759254.614294:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12311 00000100:00080000:1.0:1787759254.614296:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759254.614298:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759254.614302:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759254.614311:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655288064/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759254.614317:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759254/real 1787759254] req@ffff8c05b2d9a300 x1874588655287552/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759254.614322:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759254.614332:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759254.614338:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759254.614345:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12311 00000100:00100000:2.0:1787759254.614348:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588655287552/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759254.614351:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759254.614361:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:2.0:1787759254.614364:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787759254.614369:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787759254.614370:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759254.614375:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655287552:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787759254.614389:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655288192/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759254.614401:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655287680:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787759254.614402:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759254.614410:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787759254.614413:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759254.614419:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12311 00000100:00080000:1.0:1787759254.614424:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787759254.614425:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787759254.614434:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759254.614440:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759254.614441:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759254/real 1787759254] req@ffff8c05b2d98380 x1874588655287680/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759254.614456:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655288320/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759254.614457:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655287680/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759254.614484:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759254.614486:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759254.614488:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759254.614492:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787759254.614493:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787759254.614498:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655287680:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787759254.614501:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12311 00000100:00080000:1.0:1787759254.614508:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787759254.614509:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655287808:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787759254.614516:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759254.614523:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787759254.614529:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759254.614539:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655288448/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759254.614542:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759254/real 1787759254] req@ffff8c05b2d99880 x1874588655287808/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759254.614579:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655287808/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759254.614589:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759254.614594:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759254.614599:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655287808:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759254.614606:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655287936:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759254.614648:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759254.614660:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759254/real 1787759254] req@ffff8c05b2d98700 x1874588655287936/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759254.614673:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655287936/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759254.614690:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759254.614694:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759254.614697:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655287936:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759254.614703:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655288064:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759254.614719:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759254.614727:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759254/real 1787759254] req@ffff8c05b2d98000 x1874588655288064/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759254.614734:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655288064/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759254.614740:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759254.614742:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759254.614744:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655288064:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759254.614767:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655288192:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759254.614787:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759254.614798:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759254/real 1787759254] req@ffff8c05b2d9b100 x1874588655288192/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759254.614810:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655288192/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759254.614820:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759254.614823:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759254.614828:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655288192:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759254.614835:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655288320:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759254.614855:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759254.614867:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759254/real 1787759254] req@ffff8c05b2d9bb80 x1874588655288320/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759254.614906:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655288320/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759254.614915:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759254.614919:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759254.614923:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655288320:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759254.614929:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655288448:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759254.614951:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759254.614964:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759254/real 1787759254] req@ffff8c05b2d99f80 x1874588655288448/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759254.614976:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655288448/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759254.614985:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759254.614995:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759254.614999:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655288448:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759254.742014:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759255.190161:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759255.765992:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759256.214084:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759256.789999:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759257.238215:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759257.813948:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759258.262058:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759258.837973:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759259.286125:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787759259.734025:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759259.734035:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787759259.734039:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759259.734048:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12316 00000100:00080000:2.0:1787759259.734058:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759259.734064:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759259.734100:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73800 x1874588655288576/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759259.734126:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759259.734129:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759259.734130:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759259.734133:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12316 00000100:00080000:2.0:1787759259.734136:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759259.734138:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759259.734146:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759259.734159:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588655288704/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759259.734166:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759259.734169:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759259.734176:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787759259.734178:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655288576:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787759259.734181:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12316 00000100:00080000:2.0:1787759259.734188:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759259.734203:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759259.734209:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759259.734218:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588655288832/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759259.734229:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759259.734231:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759259.734233:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759259.734237:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12316 00000100:00080000:2.0:1787759259.734243:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759259.734247:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759259.734252:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759259.734265:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588655288960/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787759259.734267:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787759259.734281:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759259.734286:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759259.734288:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759259.734294:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12316 00000100:00080000:2.0:1787759259.734299:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759259.734304:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759259.734310:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759259.734315:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759259/real 1787759259] req@ffff8c05b2e73800 x1874588655288576/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759259.734326:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588655289088/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759259.734335:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73800 x1874588655288576/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759259.734360:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759259.734365:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759259.734366:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759259.734369:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759259.734377:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787759259.734379:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12316 00000100:00080000:2.0:1787759259.734386:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787759259.734390:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655288576:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787759259.734396:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759259.734404:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759259.734421:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588655289216/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759259.734424:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655288704:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787759259.734448:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759259.734452:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787759259.734459:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759259.734465:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12316 00000100:00080000:2.0:1787759259.734469:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787759259.734472:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787759259.734478:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759259.734484:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759259.734497:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759259/real 1787759259] req@ffff8c05b2e70e00 x1874588655288704/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759259.734502:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588655289344/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759259.734543:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787759259.734544:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588655288704/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759259.734562:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759259.734568:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759259.734582:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12316 00000100:00080000:1.0:1787759259.734587:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759259.734589:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759259.734593:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759259.734605:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655288704:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787759259.734605:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759259.734620:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759259.734622:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655288832:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759259.734648:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588655289472/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787759259.734663:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759259.734688:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759259/real 1787759259] req@ffff8c05b2e71880 x1874588655288832/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759259.734700:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588655288832/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759259.734705:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759259.734709:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759259.734713:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655288832:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759259.734718:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655288960:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759259.734733:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759259.734739:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759259/real 1787759259] req@ffff8c05b2e71c00 x1874588655288960/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759259.734746:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588655288960/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759259.734750:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759259.734752:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759259.734755:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655288960:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759259.734759:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655289088:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759259.734769:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759259.734774:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759259/real 1787759259] req@ffff8c05b2e73b80 x1874588655289088/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759259.734780:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588655289088/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759259.734784:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759259.734786:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759259.734787:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655289088:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759259.734800:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655289216:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759259.734810:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759259.734815:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759259/real 1787759259] req@ffff8c05b2e71500 x1874588655289216/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759259.734822:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588655289216/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759259.734826:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759259.734828:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759259.734830:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655289216:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759259.734833:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655289344:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759259.734842:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759259.734848:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759259/real 1787759259] req@ffff8c05b2e73480 x1874588655289344/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759259.734854:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588655289344/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759259.734859:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759259.734860:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759259.734862:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655289344:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759259.734865:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655289472:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759259.734899:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759259.734910:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759259/real 1787759259] req@ffff8c05b2e70700 x1874588655289472/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759259.734921:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588655289472/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759259.734926:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759259.734928:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759259.734931:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655289472:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759259.861992:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759260.310091:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759260.886018:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759261.334171:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759261.909999:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759262.358058:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759262.933992:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759263.382141:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759263.957968:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759264.406068:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787759264.853960:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759264.853975:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787759264.853979:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759264.853990:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12321 00000100:00080000:2.0:1787759264.854003:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759264.854011:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759264.854065:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588655289600/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759264.854099:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759264.854103:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759264.854105:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759264.854109:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12321 00000100:00080000:2.0:1787759264.854113:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759264.854117:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759264.854127:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759264.854142:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588655289728/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759264.854150:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655289600:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787759264.854156:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759264.854161:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759264.854164:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759264.854169:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12321 00000100:00080000:2.0:1787759264.854173:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759264.854177:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759264.854191:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759264.854208:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588655289856/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759264.854219:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759264.854222:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759264.854224:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759264.854229:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12321 00000100:00080000:2.0:1787759264.854232:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759264.854237:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759264.854242:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759264.854255:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588655289984/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759264.854266:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759264.854270:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759264.854271:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759264.854276:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12321 00000100:00080000:2.0:1787759264.854279:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787759264.854281:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655289728:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787759264.854287:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759264.854295:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759264.854309:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588655290112/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759264.854321:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787759264.854322:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655289856:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787759264.854328:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759264.854332:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759264.854338:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12321 00000100:00080000:2.0:1787759264.854342:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787759264.854346:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655289984:192.168.201.107@tcp:8:kworker.0 00000100:00080000:2.0:1787759264.854348:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759264.854356:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759264.854367:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655290112:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759264.854372:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588655290240/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759264.854384:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787759264.854389:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655290240:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787759264.854389:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787759264.854394:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759264.854400:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12321 00000100:00080000:2.0:1787759264.854407:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759264.854412:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759264.854417:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759264.854446:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588655290368/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759264.854463:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759264.854467:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759264.854469:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759264.854473:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12321 00000100:00080000:2.0:1787759264.854478:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759264.854483:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759264.854488:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759264.854489:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655290368:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759264.854507:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588655290496/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759264.854529:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655290496:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759264.982003:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759265.430172:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759266.005966:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759266.454013:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759267.029984:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759267.478090:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787759267.926222:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.107:988 00000400:00000100:3.0:1787759267.926275:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.107@tcp at host 192.168.201.107 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787759267.926292:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:3.0:1787759267.926332:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:3.0:1787759267.926342:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:3.0:1787759267.926351:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:3.0:1787759267.926361:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:3.0:1787759267.926371:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:3.0:1787759267.926393:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:1.0:1787759267.926399:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759264/real 1787759267] req@ffff8c05b2e72a00 x1874588655290112/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787759267.926405:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:1.0:1787759267.926431:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588655290112/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759267.926442:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759267.926447:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759267.926451:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655290112:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759267.926472:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759264/real 1787759267] req@ffff8c05b2e72d80 x1874588655289728/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759267.926484:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588655289728/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759267.926491:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759267.926494:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759267.926497:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655289728:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759267.926512:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759264/real 1787759267] req@ffff8c05b2e71180 x1874588655289856/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759267.926526:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588655289856/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759267.926533:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759267.926536:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759267.926539:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655289856:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759267.926545:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759264/real 1787759267] req@ffff8c05b2e72680 x1874588655289984/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759267.926555:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588655289984/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759267.926561:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759267.926564:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759267.926566:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655289984:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759267.926572:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759264/real 1787759267] req@ffff8c05b2e70380 x1874588655289600/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759267.926582:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588655289600/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759267.926588:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759267.926590:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759267.926593:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655289600:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787759267.926621:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759264/real 1787759267] req@ffff8c05b2e70000 x1874588655290496/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759267.926631:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588655290496/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759267.926638:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759267.926640:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759267.926643:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655290496:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759267.926649:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759264/real 1787759267] req@ffff8c05b2e70a80 x1874588655290368/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759267.926658:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588655290368/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759267.926664:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759267.926667:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759267.926670:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655290368:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759267.926677:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759264/real 1787759267] req@ffff8c05b2e72300 x1874588655290240/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759267.926686:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588655290240/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759267.926693:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759267.926696:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759267.926699:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655290240:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759268.053982:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759268.502036:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759269.077994:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759269.526139:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787759269.974000:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759269.974011:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787759269.974016:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759269.974027:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12326 00000100:00080000:0.0:1787759269.974038:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759269.974046:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787759269.974087:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588655290624/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759269.974173:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655290624:192.168.201.107@tcp:250:kworker.0 00000800:00000100:1.0:1787759269.974247:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787759269.974285:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759269.974290:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787759269.974292:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759269.974297:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12326 00000100:00080000:0.0:1787759269.974301:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759269.974306:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759269.974317:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759269.974332:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655290752/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759269.975269:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759269/real 1787759269] req@ffff8c05a9f75c00 x1874588655290624/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759269.975295:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588655290624/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759269.975308:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759269.975313:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759269.975317:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655290624:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787759269.975360:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655290752:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759269.975397:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759269.975413:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759269/real 1787759269] req@ffff8c05a9f76300 x1874588655290752/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759269.975425:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655290752/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759269.975434:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759269.975437:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759269.975441:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655290752:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787759269.975626:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759269.975632:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787759269.975634:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759269.975639:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12326 00000100:00080000:0.0:1787759269.975644:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759269.975649:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759269.975657:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759269.975676:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588655290880/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759269.975701:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759269.975706:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787759269.975708:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759269.975712:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12326 00000100:00080000:0.0:1787759269.975717:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759269.975721:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759269.975726:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759269.975737:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655291008/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759269.975746:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759269.975749:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787759269.975750:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759269.975754:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12326 00000100:00080000:0.0:1787759269.975757:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759269.975761:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759269.975766:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759269.975775:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588655291136/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759269.975784:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759269.975787:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787759269.975789:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759269.975793:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12326 00000100:00080000:0.0:1787759269.975796:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759269.975800:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759269.975805:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759269.975815:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655291264/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759269.975823:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759269.975826:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787759269.975827:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759269.975830:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12326 00000100:00080000:0.0:1787759269.975833:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759269.975837:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759269.975841:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759269.975851:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655291392/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759269.975859:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759269.975862:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787759269.975863:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759269.975867:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12326 00000100:00080000:0.0:1787759269.975870:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759269.975874:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759269.975894:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759269.975905:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655291520/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759269.975974:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655290880:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759269.976002:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759269.976015:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759269/real 1787759269] req@ffff8c05a9f76a00 x1874588655290880/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759269.976029:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588655290880/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759269.976037:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759269.976040:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759269.976043:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655290880:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759269.976048:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655291008:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759269.976067:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759269.976076:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759269/real 1787759269] req@ffff8c05a9f77800 x1874588655291008/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759269.976086:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655291008/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759269.976094:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759269.976097:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759269.976101:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655291008:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759269.976106:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655291136:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759269.976125:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759269.976135:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759269/real 1787759269] req@ffff8c05a9f77b80 x1874588655291136/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759269.976153:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588655291136/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759269.976162:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759269.976165:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759269.976168:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655291136:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759269.976173:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655291264:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759269.976192:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759269.976202:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759269/real 1787759269] req@ffff8c05a9f76680 x1874588655291264/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759269.976210:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655291264/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759269.976218:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759269.976221:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759269.976224:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655291264:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759269.976229:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655291392:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759269.976248:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759269.976256:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759269/real 1787759269] req@ffff8c05a9f74380 x1874588655291392/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759269.976266:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655291392/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759269.976278:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759269.976281:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759269.976283:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655291392:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759269.976288:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655291520:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759269.976301:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759269.976310:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759269/real 1787759269] req@ffff8c05a9f74a80 x1874588655291520/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759269.976319:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655291520/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759269.976326:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759269.976328:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759269.976331:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655291520:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759270.101982:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759270.550017:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759271.125991:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759271.574144:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759272.149974:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759272.598105:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759273.174009:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759273.622112:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759274.198018:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759274.646072:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787759275.094060:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759275.094071:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787759275.094074:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759275.094084:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12331 00000100:00080000:1.0:1787759275.094095:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759275.094101:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759275.094137:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655291648/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759275.094166:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759275.094169:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759275.094170:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759275.094173:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12331 00000100:00080000:1.0:1787759275.094175:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759275.094177:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759275.094184:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759275.094192:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655291776/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759275.094198:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759275.094201:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759275.094202:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759275.094205:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12331 00000100:00080000:1.0:1787759275.094207:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759275.094209:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759275.094212:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759275.094219:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655291904/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759275.094225:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759275.094229:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787759275.094230:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655291648:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787759275.094231:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759275.094235:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12331 00000100:00080000:1.0:1787759275.094245:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759275.094260:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759275.094264:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759275.094273:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655292032/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759275.094280:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759275.094282:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759275.094283:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759275.094285:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12331 00000100:00080000:1.0:1787759275.094288:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759275.094291:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759275.094293:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759275.094300:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655292160/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759275.094305:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759275.094308:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759275.094308:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759275.094311:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12331 00000100:00080000:1.0:1787759275.094312:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759275.094315:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:2.0:1787759275.094318:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:1.0:1787759275.094319:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759275.094328:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655292288/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759275.094337:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759275.094339:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787759275.094340:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759275.094343:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12331 00000100:00080000:1.0:1787759275.094344:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759275.094347:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759275.094350:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759275.094356:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655292416/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759275.094362:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759275.094365:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787759275.094365:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759275/real 1787759275] req@ffff8c05b2d9aa00 x1874588655291648/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759275.094368:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759275.094375:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12331 00000100:00080000:1.0:1787759275.094379:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759275.094383:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759275.094387:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759275.094391:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655291648/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759275.094395:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655292544/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759275.094410:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759275.094414:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759275.094418:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655291648:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787759275.094424:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655291776:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759275.094440:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759275.094447:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759275/real 1787759275] req@ffff8c05b2d9b480 x1874588655291776/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759275.094454:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655291776/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759275.094459:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759275.094461:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759275.094462:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655291776:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759275.094482:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655291904:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759275.094494:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759275.094500:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759275/real 1787759275] req@ffff8c05b2d99f80 x1874588655291904/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759275.094506:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655291904/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759275.094511:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759275.094513:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759275.094514:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655291904:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759275.094518:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655292032:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759275.094527:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759275.094531:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759275/real 1787759275] req@ffff8c05b2d9bb80 x1874588655292032/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759275.094545:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655292032/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759275.094550:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759275.094552:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759275.094553:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655292032:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759275.094556:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655292160:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759275.094567:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759275.094572:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759275/real 1787759275] req@ffff8c05b2d9b100 x1874588655292160/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759275.094587:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655292160/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759275.094595:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759275.094598:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759275.094601:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655292160:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759275.094606:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655292288:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759275.094624:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759275.094634:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759275/real 1787759275] req@ffff8c05b2d98000 x1874588655292288/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759275.094645:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655292288/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759275.094652:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759275.094655:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759275.094658:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655292288:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759275.094663:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655292416:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759275.094691:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759275.094701:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759275/real 1787759275] req@ffff8c05b2d98700 x1874588655292416/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759275.094712:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655292416/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759275.094725:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759275.094728:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759275.094731:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655292416:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759275.094737:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655292544:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759275.094754:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759275.094763:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759275/real 1787759275] req@ffff8c05b2d99880 x1874588655292544/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759275.094773:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655292544/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759275.094780:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759275.094783:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759275.094786:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655292544:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759275.221954:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759275.670094:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759276.246006:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759276.694080:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759277.269958:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759277.718109:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759278.293985:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759278.742157:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759279.318016:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759279.766096:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787759280.213997:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759280.214009:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787759280.214014:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759280.214026:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12336 00000100:00080000:1.0:1787759280.214038:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759280.214046:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759280.214078:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655292672/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759280.214106:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759280.214114:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759280.214116:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759280.214121:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12336 00000100:00080000:1.0:1787759280.214126:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759280.214131:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759280.214142:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759280.214154:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588655292800/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759280.214170:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655292672:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787759280.214183:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759280.214186:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759280.214189:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759280.214196:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12336 00000100:00080000:1.0:1787759280.214200:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759280.214205:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759280.214212:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759280.214229:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655292928/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759280.214245:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759280.214250:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:2.0:1787759280.214250:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787759280.214254:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759280.214263:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12336 00000100:00080000:1.0:1787759280.214269:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759280.214274:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759280.214281:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759280.214295:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655293056/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759280.214305:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759280.214309:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759280.214312:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787759280.214319:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759280/real 1787759280] req@ffff8c05b2d98380 x1874588655292672/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759280.214321:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12336 00000100:00080000:1.0:1787759280.214329:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759280.214337:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759280.214346:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759280.214352:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655292672/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759280.214360:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655293184/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759280.214365:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759280.214373:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759280.214376:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787759280.214378:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759280.214383:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787759280.214387:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655292672:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787759280.214392:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12336 00000100:00080000:1.0:1787759280.214400:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759280.214407:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759280.214413:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759280.214424:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655292800:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759280.214427:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655293312/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759280.214437:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759280.214440:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787759280.214442:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759280.214446:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12336 00000100:00080000:1.0:1787759280.214452:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759280.214459:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:2.0:1787759280.214459:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:1.0:1787759280.214468:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759280.214483:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759280/real 1787759280] req@ffff8c05b2d9a300 x1874588655292800/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759280.214485:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655293440/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759280.214514:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759280.214517:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759280.214521:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787759280.214528:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588655292800/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759280.214529:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12336 00000100:00080000:1.0:1787759280.214537:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759280.214543:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759280.214544:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:2.0:1787759280.214553:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000008:00100000:1.0:1787759280.214554:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759280.214560:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655292800:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759280.214568:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655292928:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759280.214570:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655293568/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787759280.214592:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759280.214608:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759280/real 1787759280] req@ffff8c05b2d99880 x1874588655292928/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759280.214620:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655292928/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759280.214628:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759280.214630:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759280.214633:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655292928:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759280.214638:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655293056:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759280.214655:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759280.214665:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759280/real 1787759280] req@ffff8c05b2d98700 x1874588655293056/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759280.214676:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655293056/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759280.214683:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759280.214686:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759280.214689:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655293056:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759280.214694:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655293184:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759280.214710:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759280.214720:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759280/real 1787759280] req@ffff8c05b2d98000 x1874588655293184/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759280.214731:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655293184/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759280.214738:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759280.214741:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759280.214744:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655293184:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759280.214757:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655293312:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759280.214775:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759280.214781:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759280/real 1787759280] req@ffff8c05b2d9b100 x1874588655293312/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759280.214788:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655293312/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759280.214797:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759280.214799:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759280.214800:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655293312:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759280.214803:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655293440:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759280.214812:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759280.214817:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759280/real 1787759280] req@ffff8c05b2d9bb80 x1874588655293440/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759280.214823:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655293440/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759280.214828:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759280.214830:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759280.214831:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655293440:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759280.214834:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655293568:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759280.214842:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759280.214847:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759280/real 1787759280] req@ffff8c05b2d99f80 x1874588655293568/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759280.214853:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655293568/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759280.214858:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759280.214859:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759280.214861:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655293568:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759280.341984:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759280.790095:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759281.365972:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759281.814076:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759282.389992:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759282.838048:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759283.413992:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759283.862138:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759284.437992:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759284.886098:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787759285.334013:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759285.334024:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787759285.334028:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759285.334038:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12341 00000100:00080000:1.0:1787759285.334048:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759285.334055:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759285.334107:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655293696/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759285.334128:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759285.334131:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759285.334132:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759285.334135:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12341 00000100:00080000:1.0:1787759285.334137:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759285.334140:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759285.334148:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759285.334157:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655293824/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759285.334163:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759285.334166:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759285.334167:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759285.334170:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12341 00000100:00080000:1.0:1787759285.334172:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759285.334177:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759285.334177:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655293696:192.168.201.107@tcp:250:kworker.0 00000008:00100000:1.0:1787759285.334183:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759285.334192:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655293952/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759285.334198:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759285.334201:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759285.334202:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759285.334204:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12341 00000100:00080000:1.0:1787759285.334206:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759285.334209:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759285.334212:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759285.334219:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655294080/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759285.334224:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759285.334227:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759285.334228:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759285.334230:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12341 00000100:00080000:1.0:1787759285.334232:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759285.334234:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759285.334237:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787759285.334242:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759285.334244:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655294208/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759285.334255:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759285.334257:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759285.334259:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759285.334261:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12341 00000100:00080000:1.0:1787759285.334264:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759285.334266:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759285.334268:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759285.334274:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655294336/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759285.334279:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759285.334281:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787759285.334283:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787759285.334285:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759285/real 1787759285] req@ffff8c05b2d9b480 x1874588655293696/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759285.334289:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12341 00000100:00080000:1.0:1787759285.334295:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759285.334301:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759285.334306:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759285.334306:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655293696/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759285.334318:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759285.334322:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759285.334323:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655294464/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759285.334326:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655293696:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787759285.334336:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655293824:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787759285.334340:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759285.334343:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759285.334345:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759285.334348:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12341 00000100:00080000:1.0:1787759285.334352:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787759285.334358:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787759285.334359:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759285.334366:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759285.334373:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759285/real 1787759285] req@ffff8c05b2d9aa00 x1874588655293824/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759285.334381:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655294592/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759285.334392:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655293824/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759285.334404:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759285.334407:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759285.334411:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655293824:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759285.334434:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655293952:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759285.334448:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759285.334456:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759285/real 1787759285] req@ffff8c05b2d99f80 x1874588655293952/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759285.334463:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655293952/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759285.334467:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759285.334469:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759285.334471:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655293952:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759285.334474:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655294080:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759285.334492:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759285.334500:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759285/real 1787759285] req@ffff8c05b2d9bb80 x1874588655294080/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759285.334511:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655294080/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759285.334518:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759285.334521:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759285.334524:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655294080:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759285.334529:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655294208:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759285.334546:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759285.334556:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759285/real 1787759285] req@ffff8c05b2d9b100 x1874588655294208/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759285.334566:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655294208/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759285.334574:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759285.334577:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759285.334580:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655294208:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759285.334585:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655294336:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759285.334601:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759285.334608:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759285/real 1787759285] req@ffff8c05b2d98000 x1874588655294336/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759285.334618:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655294336/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759285.334625:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759285.334628:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759285.334631:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655294336:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759285.334637:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655294464:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759285.334655:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759285.334662:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759285/real 1787759285] req@ffff8c05b2d98700 x1874588655294464/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759285.334670:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655294464/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759285.334678:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759285.334680:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759285.334684:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655294464:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759285.334690:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655294592:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759285.334707:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759285.334716:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759285/real 1787759285] req@ffff8c05b2d99880 x1874588655294592/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759285.334724:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655294592/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759285.334728:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759285.334730:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759285.334732:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655294592:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759285.461967:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759285.910087:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759286.486006:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759286.934096:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759287.509967:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759287.958068:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759288.533973:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759288.982108:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759289.558021:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759290.006120:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787759290.454020:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759290.454041:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787759290.454047:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759290.454058:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12346 00000100:00080000:3.0:1787759290.454071:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759290.454078:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787759290.454120:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655294720/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759290.454149:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759290.454153:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759290.454155:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759290.454160:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12346 00000100:00080000:3.0:1787759290.454164:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759290.454167:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759290.454178:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759290.454192:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655294848/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759290.454202:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759290.454206:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759290.454208:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759290.454213:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12346 00000100:00080000:3.0:1787759290.454217:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759290.454221:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759290.454231:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655294720:192.168.201.107@tcp:250:kworker.0 00000008:00100000:3.0:1787759290.454231:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759290.454254:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655294976/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759290.454268:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759290.454273:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759290.454275:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759290.454279:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12346 00000100:00080000:3.0:1787759290.454294:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759290.454300:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:2.0:1787759290.454306:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:3.0:1787759290.454308:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759290.454325:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655295104/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759290.454337:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759290.454340:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759290.454343:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759290.454347:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12346 00000100:00080000:3.0:1787759290.454351:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759290.454355:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759290.454359:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759290.454370:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759290/real 1787759290] req@ffff8c05a9f7a300 x1874588655294720/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759290.454376:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655295232/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759290.454408:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787759290.454414:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655294720/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759290.454414:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759290.454420:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759290.454430:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12346 00000100:00080000:3.0:1787759290.454437:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759290.454445:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:2.0:1787759290.454450:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787759290.454451:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787759290.454461:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759290.454470:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655294720:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787759290.454471:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655295360/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759290.454492:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787759290.454493:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655294848:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787759290.454504:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787759290.454507:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759290.454510:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12346 00000100:00080000:3.0:1787759290.454514:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759290.454519:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759290.454523:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787759290.454535:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759290.454535:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655295488/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759290.454547:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759290.454550:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759290.454551:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759290.454554:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12346 00000100:00080000:3.0:1787759290.454556:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787759290.454562:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759290/real 1787759290] req@ffff8c05a9f79880 x1874588655294848/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759290.454563:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759290.454571:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759290.454586:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655295616/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759290.454593:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655294848/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759290.454603:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759290.454607:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759290.454609:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655294848:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759290.454634:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655294976:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759290.454658:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759290.454669:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759290/real 1787759290] req@ffff8c05a9f78700 x1874588655294976/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759290.454681:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655294976/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759290.454688:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759290.454692:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759290.454695:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655294976:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759290.454701:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655295104:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759290.454722:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759290.454731:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759290/real 1787759290] req@ffff8c05a9f7b480 x1874588655295104/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759290.454738:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655295104/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759290.454744:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759290.454746:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759290.454748:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655295104:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759290.454752:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655295232:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759290.454764:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759290.454770:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759290/real 1787759290] req@ffff8c05a9f78380 x1874588655295232/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759290.454777:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655295232/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759290.454781:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759290.454783:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759290.454784:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655295232:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759290.454788:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655295360:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759290.454797:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759290.454802:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759290/real 1787759290] req@ffff8c05a9f78a80 x1874588655295360/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759290.454808:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655295360/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759290.454819:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759290.454821:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759290.454824:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655295360:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759290.454827:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655295488:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759290.454838:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759290.454844:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759290/real 1787759290] req@ffff8c05a9f7a680 x1874588655295488/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759290.454851:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655295488/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759290.454855:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759290.454857:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759290.454858:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655295488:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759290.454861:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655295616:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759290.454870:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759290.454909:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759290/real 1787759290] req@ffff8c05a9f7aa00 x1874588655295616/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759290.454918:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655295616/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759290.454923:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759290.454925:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759290.454927:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655295616:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759290.582023:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759291.030047:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759291.606032:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759292.054165:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759292.629969:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759293.078029:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759293.654006:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759294.102169:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759294.677974:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759295.126097:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787759295.574054:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759295.574064:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787759295.574069:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759295.574080:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12351 00000100:00080000:2.0:1787759295.574092:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759295.574099:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759295.574147:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588655295744/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759295.574173:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759295.574176:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759295.574177:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759295.574180:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12351 00000100:00080000:2.0:1787759295.574182:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759295.574185:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759295.574193:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759295.574226:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588655295872/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759295.574234:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759295.574237:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787759295.574238:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655295744:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787759295.574239:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759295.574245:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12351 00000100:00080000:2.0:1787759295.574249:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759295.574252:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759295.574255:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759295.574264:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588655296000/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759295.574275:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759295.574277:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759295.574278:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759295.574281:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12351 00000100:00080000:2.0:1787759295.574283:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759295.574285:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759295.574288:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759295.574294:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588655296128/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759295.574300:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759295.574303:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759295.574304:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759295.574305:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12351 00000100:00080000:2.0:1787759295.574307:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759295.574309:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759295.574312:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759295.574317:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588655296256/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759295.574322:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759295.574324:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759295.574325:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759295.574327:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12351 00000100:00080000:2.0:1787759295.574329:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759295.574332:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759295.574335:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759295.574340:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588655296384/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759295.574344:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759295.574346:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787759295.574347:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759295.574349:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12351 00000100:00080000:2.0:1787759295.574351:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787759295.574357:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787759295.574357:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759295.574362:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759295.574368:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588655296512/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759295.574373:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759295.574375:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759295.574376:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759295.574378:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12351 00000100:00080000:2.0:1787759295.574380:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759295.574382:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759295.574384:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759295.574391:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588655296640/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759295.574414:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759295/real 1787759295] req@ffff8c05b2e73100 x1874588655295744/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759295.574431:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588655295744/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759295.574441:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759295.574444:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759295.574448:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655295744:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787759295.574471:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655295872:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759295.574492:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759295.574503:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759295/real 1787759295] req@ffff8c05b2e72300 x1874588655295872/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759295.574519:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588655295872/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759295.574527:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759295.574530:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759295.574533:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655295872:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759295.574538:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655296000:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759295.574554:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759295.574563:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759295/real 1787759295] req@ffff8c05b2e70a80 x1874588655296000/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759295.574573:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588655296000/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759295.574579:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759295.574582:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759295.574585:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655296000:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759295.574598:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655296128:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759295.574616:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759295.574625:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759295/real 1787759295] req@ffff8c05b2e70000 x1874588655296128/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759295.574635:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588655296128/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759295.574647:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759295.574650:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759295.574653:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655296128:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759295.574658:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655296256:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759295.574675:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759295.574685:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759295/real 1787759295] req@ffff8c05b2e70380 x1874588655296256/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759295.574696:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588655296256/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759295.574703:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759295.574706:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759295.574709:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655296256:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759295.574714:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655296384:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759295.574728:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759295.574735:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759295/real 1787759295] req@ffff8c05b2e72680 x1874588655296384/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759295.574741:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588655296384/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759295.574746:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759295.574748:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759295.574750:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655296384:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759295.574754:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655296512:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759295.574763:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759295.574768:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759295/real 1787759295] req@ffff8c05b2e71180 x1874588655296512/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759295.574774:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588655296512/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759295.574778:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759295.574780:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759295.574781:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655296512:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759295.574784:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655296640:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759295.574793:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759295.574798:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759295/real 1787759295] req@ffff8c05b2e72d80 x1874588655296640/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759295.574805:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588655296640/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759295.574809:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759295.574810:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759295.574812:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655296640:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759295.702027:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759296.150165:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759296.725941:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759297.174085:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759297.749970:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759298.198170:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759298.774005:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759299.222090:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759299.797975:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759300.246155:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787759300.694051:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759300.694061:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787759300.694064:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759300.694074:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12357 00000100:00080000:1.0:1787759300.694083:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759300.694090:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759300.694125:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588655296768/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759300.694150:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759300.694152:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759300.694153:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759300.694155:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12357 00000100:00080000:1.0:1787759300.694157:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759300.694159:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759300.694166:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759300.694175:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655296896/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759300.694181:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759300.694184:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759300.694185:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759300.694187:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12357 00000100:00080000:1.0:1787759300.694188:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759300.694190:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759300.694193:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759300.694196:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655296768:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787759300.694204:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655297024/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759300.694211:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759300.694213:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759300.694214:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759300.694216:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12357 00000100:00080000:1.0:1787759300.694217:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759300.694219:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759300.694228:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759300.694237:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655297152/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759300.694243:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759300.694245:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759300.694246:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759300.694248:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12357 00000100:00080000:1.0:1787759300.694250:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759300.694253:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759300.694256:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759300.694262:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655297280/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759300.694270:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787759300.694270:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787759300.694275:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759300.694276:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759300.694281:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12357 00000100:00080000:1.0:1787759300.694283:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759300.694286:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759300.694289:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759300.694295:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655297408/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759300.694300:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759300.694301:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787759300.694302:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759300.694305:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12357 00000100:00080000:1.0:1787759300.694307:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759300.694309:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759300.694311:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759300.694318:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655297536/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759300.694324:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787759300.694325:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759300/real 1787759300] req@ffff8c05b2d9a300 x1874588655296768/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759300.694327:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759300.694329:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759300.694335:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12357 00000100:00080000:1.0:1787759300.694338:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759300.694343:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759300.694352:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759300.694360:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588655296768/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759300.694365:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655297664/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759300.694386:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759300.694393:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759300.694398:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655296768:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787759300.694408:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655296896:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759300.694433:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759300.694446:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759300/real 1787759300] req@ffff8c05b2d98380 x1874588655296896/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759300.694459:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655296896/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759300.694467:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759300.694470:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759300.694474:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655296896:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759300.694499:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655297024:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759300.694520:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759300.694533:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759300/real 1787759300] req@ffff8c05b2d99880 x1874588655297024/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759300.694544:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655297024/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759300.694552:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759300.694555:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759300.694559:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655297024:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759300.694579:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655297152:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759300.694601:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759300.694613:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759300/real 1787759300] req@ffff8c05b2d98700 x1874588655297152/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759300.694624:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655297152/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759300.694632:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759300.694635:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759300.694638:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655297152:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759300.694644:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655297280:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759300.694663:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759300.694682:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759300/real 1787759300] req@ffff8c05b2d98000 x1874588655297280/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759300.694698:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655297280/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759300.694707:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759300.694710:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759300.694713:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655297280:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759300.694718:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655297408:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759300.694739:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759300.694750:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759300/real 1787759300] req@ffff8c05b2d9b100 x1874588655297408/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759300.694762:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655297408/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759300.694769:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759300.694772:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759300.694776:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655297408:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759300.694781:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655297536:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759300.694816:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759300.694827:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759300/real 1787759300] req@ffff8c05b2d9bb80 x1874588655297536/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759300.694838:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655297536/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759300.694846:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759300.694849:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759300.694852:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655297536:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759300.694858:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655297664:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759300.694900:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759300.694912:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759300/real 1787759300] req@ffff8c05b2d99f80 x1874588655297664/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759300.694923:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655297664/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759300.694931:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759300.694934:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759300.694937:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655297664:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759300.822018:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759301.270061:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759301.846005:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759302.294095:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759302.869991:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759303.318056:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759303.893996:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759304.342137:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759304.918002:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759305.365986:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787759305.814045:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759305.814056:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787759305.814061:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759305.814072:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12362 00000100:00080000:3.0:1787759305.814083:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759305.814090:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787759305.814132:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655297792/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759305.814161:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759305.814164:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759305.814166:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759305.814170:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12362 00000100:00080000:3.0:1787759305.814173:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759305.814177:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759305.814185:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759305.814199:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655297920/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759305.814209:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787759305.814210:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655297792:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787759305.814214:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759305.814218:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759305.814222:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12362 00000100:00080000:3.0:1787759305.814224:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759305.814227:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759305.814232:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759305.814242:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655298048/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759305.814249:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759305.814252:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759305.814254:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759305.814257:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12362 00000100:00080000:3.0:1787759305.814261:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759305.814264:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759305.814268:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759305.814277:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655298176/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787759305.814283:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787759305.814291:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759305.814294:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759305.814295:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759305.814299:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12362 00000100:00080000:3.0:1787759305.814302:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759305.814305:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759305.814309:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759305.814319:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655298304/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759305.814326:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759305.814329:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759305.814330:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759305.814333:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12362 00000100:00080000:3.0:1787759305.814335:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759305.814338:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759305.814341:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759305.814352:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759305/real 1787759305] req@ffff8c05a9f7aa00 x1874588655297792/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759305.814353:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655298432/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759305.814380:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759305.814383:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787759305.814385:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759305.814388:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12362 00000100:00080000:3.0:1787759305.814392:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787759305.814396:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655297792/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759305.814397:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759305.814403:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787759305.814409:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759305.814414:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759305.814418:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655298560/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759305.814422:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655297792:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787759305.814431:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759305.814434:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759305.814435:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759305.814440:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12362 00000100:00080000:3.0:1787759305.814442:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759305.814445:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759305.814448:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759305.814456:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655298688/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759305.814461:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655297920:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759305.814490:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759305.814503:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759305/real 1787759305] req@ffff8c05a9f7a680 x1874588655297920/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759305.814514:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655297920/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759305.814522:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759305.814526:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759305.814529:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655297920:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759305.814534:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655298048:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759305.814551:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759305.814561:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759305/real 1787759305] req@ffff8c05a9f78a80 x1874588655298048/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759305.814570:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655298048/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759305.814577:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759305.814580:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759305.814582:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655298048:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759305.814587:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655298176:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759305.814605:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759305.814615:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759305/real 1787759305] req@ffff8c05a9f78380 x1874588655298176/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759305.814626:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655298176/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759305.814632:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759305.814634:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759305.814636:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655298176:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759305.814639:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655298304:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759305.814652:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759305.814666:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759305/real 1787759305] req@ffff8c05a9f7b480 x1874588655298304/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759305.814673:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655298304/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759305.814677:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759305.814679:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759305.814681:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655298304:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759305.814684:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655298432:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759305.814694:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759305.814699:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759305/real 1787759305] req@ffff8c05a9f78700 x1874588655298432/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759305.814705:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655298432/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759305.814724:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759305.814727:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759305.814731:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655298432:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759305.814736:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655298560:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759305.814756:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759305.814766:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759305/real 1787759305] req@ffff8c05a9f79880 x1874588655298560/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759305.814774:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655298560/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759305.814779:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759305.814780:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759305.814782:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655298560:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759305.814795:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655298688:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759305.814807:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759305.814813:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759305/real 1787759305] req@ffff8c05a9f7a300 x1874588655298688/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759305.814819:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655298688/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759305.814824:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759305.814826:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759305.814827:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655298688:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759305.942005:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759306.390185:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759306.965980:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759307.416287:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759307.989990:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759308.438195:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759309.014000:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759309.462086:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759310.038076:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759310.486162:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787759310.935030:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759310.935046:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787759310.935050:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759310.935059:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12367 00000100:00080000:1.0:1787759310.935069:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759310.935075:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759310.935112:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655298816/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759310.935131:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759310.935133:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759310.935134:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759310.935137:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12367 00000100:00080000:1.0:1787759310.935139:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759310.935141:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759310.935148:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759310.935156:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655298944/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759310.935162:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759310.935165:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759310.935166:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759310.935168:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12367 00000100:00080000:1.0:1787759310.935170:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759310.935173:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759310.935176:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759310.935181:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655299072/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759310.935186:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655298816:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787759310.935189:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759310.935195:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759310.935196:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759310.935199:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12367 00000100:00080000:1.0:1787759310.935201:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759310.935203:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759310.935206:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759310.935213:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655299200/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759310.935218:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759310.935221:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759310.935222:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759310.935224:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12367 00000100:00080000:1.0:1787759310.935226:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759310.935228:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759310.935232:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759310.935237:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655299328/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759310.935243:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759310.935245:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759310.935246:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759310.935248:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12367 00000100:00080000:1.0:1787759310.935250:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759310.935254:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:2.0:1787759310.935256:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:1.0:1787759310.935257:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759310.935266:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655299456/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759310.935270:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759310.935272:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787759310.935274:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759310.935276:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12367 00000100:00080000:1.0:1787759310.935279:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759310.935281:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759310.935283:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759310.935289:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655299584/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759310.935294:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759310.935295:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759310.935297:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759310.935299:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12367 00000100:00080000:1.0:1787759310.935301:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759310.935303:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759310.935306:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759310.935317:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655299712/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759310.935318:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759310/real 1787759310] req@ffff8c05b2d9aa00 x1874588655298816/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759310.935343:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655298816/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759310.935354:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759310.935358:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759310.935362:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655298816:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787759310.935367:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655298944:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759310.935385:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759310.935393:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759310/real 1787759310] req@ffff8c05b2d9b480 x1874588655298944/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759310.935400:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655298944/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759310.935405:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759310.935407:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759310.935409:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655298944:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759310.935441:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655299072:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759310.935453:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759310.935460:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759310/real 1787759310] req@ffff8c05b2d99f80 x1874588655299072/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759310.935467:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655299072/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759310.935475:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759310.935477:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759310.935479:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655299072:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759310.935482:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655299200:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759310.935491:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759310.935495:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759310/real 1787759310] req@ffff8c05b2d9bb80 x1874588655299200/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759310.935501:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655299200/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759310.935506:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759310.935508:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759310.935509:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655299200:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759310.935512:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655299328:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759310.935520:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759310.935525:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759310/real 1787759310] req@ffff8c05b2d9b100 x1874588655299328/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759310.935532:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655299328/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759310.935536:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759310.935537:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759310.935540:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655299328:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759310.935543:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655299456:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759310.935552:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759310.935557:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759310/real 1787759310] req@ffff8c05b2d98000 x1874588655299456/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759310.935563:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655299456/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759310.935568:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759310.935569:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759310.935571:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655299456:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759310.935573:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655299584:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759310.935582:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759310.935586:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759310/real 1787759310] req@ffff8c05b2d98700 x1874588655299584/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759310.935592:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655299584/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759310.935596:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759310.935598:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759310.935600:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655299584:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759310.935603:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655299712:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759310.935629:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759310.935641:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759310/real 1787759310] req@ffff8c05b2d99880 x1874588655299712/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759310.935657:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655299712/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759310.935664:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759310.935667:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759310.935670:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655299712:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759311.062122:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759311.510100:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759312.085963:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759312.534143:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759313.110012:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759313.558279:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759314.134019:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759314.582058:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759315.157995:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759315.606117:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787759316.054049:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759316.054063:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787759316.054067:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759316.054079:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12372 00000100:00080000:1.0:1787759316.054091:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759316.054099:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759316.054142:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655299840/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759316.054173:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759316.054177:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759316.054183:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759316.054187:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12372 00000100:00080000:1.0:1787759316.054191:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759316.054195:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759316.054203:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759316.054217:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588655299968/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759316.054228:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787759316.054234:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655299840:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787759316.054244:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759316.054247:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759316.054251:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12372 00000100:00080000:1.0:1787759316.054254:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759316.054258:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759316.054263:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759316.054277:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655300096/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759316.054287:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759316.054291:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759316.054292:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759316.054296:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12372 00000100:00080000:1.0:1787759316.054299:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759316.054302:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759316.054307:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787759316.054315:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759316.054318:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655300224/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759316.054329:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759316.054332:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759316.054334:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759316.054339:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12372 00000100:00080000:1.0:1787759316.054342:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759316.054345:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759316.054349:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759316.054357:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655300352/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759316.054366:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759316.054369:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759316.054370:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759316.054376:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12372 00000100:00100000:2.0:1787759316.054376:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759316/real 1787759316] req@ffff8c05b2d98380 x1874588655299840/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759316.054381:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759316.054388:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759316.054395:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759316.054406:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655299840/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759316.054409:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655300480/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759316.054428:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759316.054430:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759316.054434:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787759316.054436:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787759316.054439:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787759316.054440:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655299840:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787759316.054448:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12372 00000100:00080000:1.0:1787759316.054453:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759316.054457:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759316.054471:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759316.054480:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655299968:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759316.054484:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655300608/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759316.054496:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759316.054500:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759316.054502:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759316.054505:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12372 00000100:00080000:1.0:1787759316.054509:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787759316.054510:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787759316.054515:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759316.054520:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759316.054543:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655300736/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759316.054549:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759316/real 1787759316] req@ffff8c05b2d9a300 x1874588655299968/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759316.054566:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588655299968/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759316.054581:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759316.054584:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759316.054588:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655299968:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759316.054606:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655300096:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759316.054628:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759316.054642:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759316/real 1787759316] req@ffff8c05b2d99880 x1874588655300096/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759316.054665:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655300096/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759316.054674:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759316.054677:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759316.054680:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655300096:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759316.054686:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655300224:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759316.054708:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759316.054720:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759316/real 1787759316] req@ffff8c05b2d98700 x1874588655300224/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759316.054731:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655300224/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759316.054739:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759316.054742:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759316.054753:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655300224:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759316.054760:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655300352:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759316.054783:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759316.054797:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759316/real 1787759316] req@ffff8c05b2d98000 x1874588655300352/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759316.054808:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655300352/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759316.054816:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759316.054820:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759316.054825:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655300352:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759316.054831:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655300480:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759316.054851:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759316.054861:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759316/real 1787759316] req@ffff8c05b2d9b100 x1874588655300480/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759316.054871:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655300480/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759316.054910:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759316.054914:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759316.054917:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655300480:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759316.054937:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655300608:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759316.054961:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759316.054975:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759316/real 1787759316] req@ffff8c05b2d9bb80 x1874588655300608/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759316.054988:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655300608/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759316.054997:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759316.055000:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759316.055003:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655300608:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759316.055010:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655300736:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759316.055031:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759316.055044:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759316/real 1787759316] req@ffff8c05b2d99f80 x1874588655300736/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759316.055058:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655300736/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759316.055065:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759316.055068:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759316.055072:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655300736:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759316.181992:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759316.630173:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759317.206045:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759317.654446:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759318.230026:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759318.678236:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759319.254033:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759319.702185:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759320.277977:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759320.726091:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787759321.174491:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759321.174503:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787759321.174507:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759321.174518:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12377 00000100:00080000:0.0:1787759321.174533:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759321.174541:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787759321.174623:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655300864/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759321.174648:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759321.174652:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787759321.174654:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759321.174657:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12377 00000100:00080000:0.0:1787759321.174661:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759321.174665:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759321.174674:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759321.174687:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655300992/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759321.174696:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655300864:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787759321.174711:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759321.174715:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787759321.174716:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759321.174720:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12377 00000100:00080000:0.0:1787759321.174723:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759321.174726:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759321.174731:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759321.174742:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655301120/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759321.174751:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759321.174754:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787759321.174756:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759321.174759:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12377 00000100:00080000:0.0:1787759321.174763:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759321.174767:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759321.174772:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787759321.174773:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759321.174785:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588655301248/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759321.174795:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759321.174799:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787759321.174800:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759321.174804:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12377 00000100:00080000:0.0:1787759321.174807:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759321.174811:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759321.174820:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759321.174826:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759321/real 1787759321] req@ffff8c05a9f74a80 x1874588655300864/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759321.174832:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655301376/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759321.174849:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759321.174852:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787759321.174854:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787759321.174858:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655300864/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759321.174860:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12377 00000100:00080000:0.0:1787759321.174865:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759321.174871:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759321.174896:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759321.174906:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588655301504/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759321.174913:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759321.174916:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759321.174920:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787759321.174922:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787759321.174923:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759321.174931:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12377 00000100:00100000:2.0:1787759321.174932:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655300864:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787759321.174935:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759321.174941:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759321.174946:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759321.174956:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655301632/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759321.174964:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655300992:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787759321.174980:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759321.174983:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787759321.174986:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759321.174990:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12377 00000100:00080000:0.0:1787759321.174994:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759321.174999:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:2.0:1787759321.175000:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:0.0:1787759321.175005:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759321.175020:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759321/real 1787759321] req@ffff8c05a9f74380 x1874588655300992/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759321.175021:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588655301760/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759321.175054:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655300992/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759321.175063:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759321.175068:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759321.175071:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655300992:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759321.175078:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655301120:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759321.175097:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759321.175107:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759321/real 1787759321] req@ffff8c05a9f76680 x1874588655301120/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759321.175119:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655301120/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759321.175127:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759321.175130:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759321.175134:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655301120:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759321.175140:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655301248:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759321.175159:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759321.175169:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759321/real 1787759321] req@ffff8c05a9f77b80 x1874588655301248/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759321.175180:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588655301248/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759321.175187:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759321.175191:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759321.175193:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655301248:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759321.175199:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655301376:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759321.175217:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759321.175228:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759321/real 1787759321] req@ffff8c05a9f77800 x1874588655301376/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759321.175239:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655301376/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759321.175247:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759321.175250:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759321.175253:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655301376:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759321.175257:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655301504:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759321.175276:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759321.175286:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759321/real 1787759321] req@ffff8c05a9f76a00 x1874588655301504/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759321.175296:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588655301504/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759321.175303:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759321.175305:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759321.175308:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655301504:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759321.175325:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655301632:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759321.175344:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759321.175354:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759321/real 1787759321] req@ffff8c05a9f76300 x1874588655301632/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759321.175365:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655301632/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759321.175373:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759321.175376:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759321.175379:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655301632:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759321.175385:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655301760:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759321.175401:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759321.175411:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759321/real 1787759321] req@ffff8c05a9f75c00 x1874588655301760/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759321.175422:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588655301760/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759321.175436:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759321.175455:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759321.175458:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655301760:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759321.301992:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759321.750358:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759322.325986:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759322.774191:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759323.349993:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759323.798091:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759324.373968:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759324.822163:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759325.398022:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759325.846103:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787759326.294031:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759326.294046:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787759326.294050:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759326.294061:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12382 00000100:00080000:3.0:1787759326.294072:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759326.294080:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787759326.294118:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655301888/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759326.294149:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759326.294153:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759326.294155:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759326.294159:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12382 00000100:00080000:3.0:1787759326.294162:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759326.294166:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759326.294174:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759326.294199:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655302016/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759326.294210:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759326.294213:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759326.294216:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759326.294224:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12382 00000100:00100000:2.0:1787759326.294225:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655301888:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787759326.294231:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759326.294240:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759326.294245:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759326.294262:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655302144/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759326.294273:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759326.294277:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759326.294278:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759326.294282:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12382 00000100:00080000:3.0:1787759326.294303:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759326.294309:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759326.294317:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787759326.294319:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759326.294336:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655302272/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759326.294359:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759326.294362:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759326.294365:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759326.294369:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12382 00000100:00080000:3.0:1787759326.294373:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759326.294378:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759326.294384:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759326.294387:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759326/real 1787759326] req@ffff8c05a9f79500 x1874588655301888/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759326.294402:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655302400/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759326.294407:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655301888/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759326.294420:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759326.294427:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759326.294429:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759326.294430:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759326.294436:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787759326.294443:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12382 00000100:00100000:2.0:1787759326.294444:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655301888:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787759326.294454:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759326.294461:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759326.294468:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759326.294476:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655302016:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759326.294483:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655302528/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759326.294497:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787759326.294504:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787759326.294508:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787759326.294512:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759326.294518:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12382 00000100:00100000:2.0:1787759326.294521:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759326/real 1787759326] req@ffff8c05a9f7b800 x1874588655302016/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759326.294529:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759326.294541:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759326.294542:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655302016/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787759326.294558:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787759326.294559:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759326.294571:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759326.294581:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655302016:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759326.294581:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655302656/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759326.294604:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655302144:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787759326.294622:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759326.294626:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759326.294629:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787759326.294636:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787759326.294641:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12382 00000100:00080000:3.0:1787759326.294647:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787759326.294649:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759326/real 1787759326] req@ffff8c05a9f78e00 x1874588655302144/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759326.294659:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759326.294666:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655302144/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787759326.294668:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787759326.294686:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759326.294689:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759326.294702:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655302144:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759326.294702:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655302784/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759326.294719:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655302272:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759326.294753:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759326.294766:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759326/real 1787759326] req@ffff8c05a9f7ad80 x1874588655302272/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759326.294785:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655302272/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759326.294795:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759326.294798:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759326.294802:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655302272:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759326.294808:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655302400:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759326.294823:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759326.294830:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759326/real 1787759326] req@ffff8c05a9f7b100 x1874588655302400/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759326.294837:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655302400/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759326.294842:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759326.294844:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759326.294845:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655302400:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759326.294857:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655302528:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759326.294867:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759326.294872:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759326/real 1787759326] req@ffff8c05a9f79f80 x1874588655302528/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759326.294910:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655302528/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759326.294915:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759326.294917:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759326.294918:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655302528:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759326.294922:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655302656:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759326.294934:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759326.294940:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759326/real 1787759326] req@ffff8c05a9f7bb80 x1874588655302656/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759326.294946:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655302656/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759326.294955:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759326.294957:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759326.294959:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655302656:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759326.294962:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655302784:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759326.294970:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759326.294975:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759326/real 1787759326] req@ffff8c05a9f79180 x1874588655302784/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759326.294981:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655302784/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759326.294986:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759326.294988:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759326.294990:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655302784:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759326.422003:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759326.870124:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759327.445997:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759327.894036:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759328.470085:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759328.918131:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759329.494188:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759329.942049:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759330.519090:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759330.966343:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787759331.415290:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759331.415303:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787759331.415307:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759331.415319:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12387 00000100:00080000:1.0:1787759331.415330:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759331.415337:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759331.415375:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655302912/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759331.415411:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759331.415415:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759331.415417:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759331.415421:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12387 00000100:00080000:1.0:1787759331.415425:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759331.415429:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759331.415437:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759331.415451:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655303040/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759331.415461:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759331.415464:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759331.415465:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759331.415469:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12387 00000100:00080000:1.0:1787759331.415471:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759331.415475:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759331.415478:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759331.415487:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655303168/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759331.415495:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759331.415498:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759331.415500:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759331.415503:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12387 00000100:00080000:1.0:1787759331.415506:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759331.415509:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759331.415514:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759331.415522:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655303296/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759331.415531:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759331.415534:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759331.415535:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759331.415538:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12387 00000100:00080000:1.0:1787759331.415541:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759331.415545:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759331.415549:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759331.415556:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655303424/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759331.415564:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759331.415567:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759331.415569:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759331.415572:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12387 00000100:00080000:1.0:1787759331.415575:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759331.415578:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759331.415581:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759331.415589:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655303552/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759331.415596:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759331.415599:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787759331.415601:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759331.415604:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12387 00000100:00080000:1.0:1787759331.415607:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759331.415611:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759331.415616:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759331.415625:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655303680/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759331.415634:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759331.415636:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759331.415638:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759331.415641:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12387 00000100:00080000:1.0:1787759331.415644:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759331.415647:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759331.415651:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759331.415660:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655303808/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759331.415762:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655302912:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787759331.415866:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655303040:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759331.415930:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655303168:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759331.415950:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655303296:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759331.415968:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655303424:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759331.415987:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655303552:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759331.416010:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655303680:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759331.416026:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655303808:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759331.542185:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759331.990491:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759332.566015:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759333.014138:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759333.590014:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759334.038096:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787759334.486245:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.107:988 00000400:00000100:0.0:1787759334.486304:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.107@tcp at host 192.168.201.107 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787759334.486322:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787759334.486363:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787759334.486372:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787759334.486379:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787759334.486386:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787759334.486393:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787759334.486399:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787759334.486406:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:2.0:1787759334.486411:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759331/real 1787759334] req@ffff8c05b2d9b480 x1874588655302912/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759334.486437:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655302912/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759334.486448:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759334.486452:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759334.486455:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655302912:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787759334.486463:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759331/real 1787759334] req@ffff8c05b2d9aa00 x1874588655303040/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759334.486475:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655303040/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759334.486488:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759334.486491:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759334.486495:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655303040:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759334.486502:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759331/real 1787759334] req@ffff8c05b2d99f80 x1874588655303168/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759334.486512:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655303168/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759334.486524:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759334.486527:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759334.486530:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655303168:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759334.486536:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759331/real 1787759334] req@ffff8c05b2d9bb80 x1874588655303296/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759334.486546:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655303296/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759334.486552:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759334.486556:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759334.486558:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655303296:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759334.486563:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759331/real 1787759334] req@ffff8c05b2d9b100 x1874588655303424/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759334.486573:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655303424/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759334.486593:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759334.486597:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759334.486600:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655303424:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759334.486606:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759331/real 1787759334] req@ffff8c05b2d98000 x1874588655303552/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759334.486616:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655303552/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759334.486622:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759334.486625:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759334.486628:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655303552:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759334.486634:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759331/real 1787759334] req@ffff8c05b2d98700 x1874588655303680/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759334.486644:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655303680/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759334.486651:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759334.486654:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759334.486656:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655303680:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759334.486662:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759331/real 1787759334] req@ffff8c05b2d99880 x1874588655303808/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759334.486673:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655303808/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759334.486680:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759334.486683:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759334.486686:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655303808:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759334.614024:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759335.062481:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759335.638485:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759336.086135:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787759336.533993:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759336.534005:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787759336.534009:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759336.534020:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12392 00000100:00080000:1.0:1787759336.534031:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759336.534038:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759336.534070:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588655303936/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759336.534095:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759336.534097:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759336.534098:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759336.534101:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12392 00000100:00080000:1.0:1787759336.534103:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759336.534105:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759336.534112:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759336.534120:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655304064/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759336.534126:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759336.534129:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759336.534130:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759336.534132:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12392 00000100:00080000:1.0:1787759336.534133:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759336.534136:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759336.534139:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759336.534148:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655304192/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759336.534158:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655303936:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787759336.534159:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759336.534177:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759336.534179:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759336.534184:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12392 00000100:00080000:1.0:1787759336.534188:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759336.534192:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759336.534197:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759336.534209:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655304320/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759336.534217:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759336.534221:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759336.534222:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759336.534226:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12392 00000100:00080000:1.0:1787759336.534229:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759336.534234:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759336.534239:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759336.534251:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655304448/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787759336.534252:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787759336.534263:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759336.534266:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759336.534268:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759336.534273:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12392 00000100:00080000:1.0:1787759336.534277:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759336.534280:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759336.534285:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759336.534296:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655304576/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759336.534301:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759336/real 1787759336] req@ffff8c05b2d9a300 x1874588655303936/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759336.534315:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759336.534320:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787759336.534322:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759336.534327:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12392 00000100:00080000:1.0:1787759336.534330:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787759336.534332:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588655303936/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759336.534337:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759336.534345:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787759336.534350:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759336.534358:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759336.534359:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655304704/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759336.534366:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655303936:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787759336.534374:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759336.534379:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787759336.534380:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655304064:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787759336.534382:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759336.534390:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12392 00000100:00080000:1.0:1787759336.534393:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759336.534397:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759336.534401:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787759336.534411:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759336.534415:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655304832/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759336.534427:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759336/real 1787759336] req@ffff8c05b2d98380 x1874588655304064/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759336.534441:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655304064/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759336.534450:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759336.534453:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759336.534456:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655304064:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759336.534479:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655304192:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759336.534496:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759336.534505:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759336/real 1787759336] req@ffff8c05b2d99880 x1874588655304192/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759336.534524:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655304192/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759336.534537:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759336.534540:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759336.534543:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655304192:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759336.534549:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655304320:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759336.534566:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759336.534576:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759336/real 1787759336] req@ffff8c05b2d98700 x1874588655304320/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759336.534585:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655304320/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759336.534592:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759336.534595:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759336.534598:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655304320:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759336.534609:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655304448:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759336.534625:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759336.534635:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759336/real 1787759336] req@ffff8c05b2d98000 x1874588655304448/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759336.534644:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655304448/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759336.534651:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759336.534654:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759336.534658:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655304448:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759336.534663:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655304576:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759336.534680:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759336.534690:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759336/real 1787759336] req@ffff8c05b2d9b100 x1874588655304576/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759336.534699:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655304576/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759336.534706:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759336.534709:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759336.534712:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655304576:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759336.534717:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655304704:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759336.534731:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759336.534739:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759336/real 1787759336] req@ffff8c05b2d9bb80 x1874588655304704/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759336.534749:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655304704/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759336.534755:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759336.534758:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759336.534761:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655304704:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759336.534766:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655304832:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759336.534781:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759336.534790:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759336/real 1787759336] req@ffff8c05b2d99f80 x1874588655304832/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759336.534799:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655304832/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759336.534806:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759336.534809:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759336.534811:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655304832:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759336.664509:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759337.110224:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759337.686005:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759338.134116:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759338.710026:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759339.158139:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759339.734017:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759340.182062:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759340.757997:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759341.206355:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787759341.654020:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759341.654031:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787759341.654036:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759341.654055:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12398 00000100:00080000:1.0:1787759341.654066:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759341.654073:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759341.654127:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655304960/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759341.654162:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759341.654165:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759341.654167:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759341.654170:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12398 00000100:00080000:1.0:1787759341.654174:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759341.654178:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759341.654187:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759341.654201:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655305088/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759341.654210:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759341.654214:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759341.654215:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759341.654219:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12398 00000100:00080000:1.0:1787759341.654224:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787759341.654225:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655304960:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787759341.654230:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759341.654238:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759341.654247:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655305216/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759341.654262:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759341.654265:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759341.654267:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759341.654270:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12398 00000100:00080000:1.0:1787759341.654274:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759341.654278:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759341.654282:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787759341.654300:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759341.654302:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655305344/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759341.654312:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759341.654315:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759341.654317:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759341.654320:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12398 00000100:00080000:1.0:1787759341.654324:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759341.654328:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759341.654333:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759341.654341:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655305472/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759341.654350:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759341/real 1787759341] req@ffff8c05b2d9aa00 x1874588655304960/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759341.654366:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759341.654371:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759341.654373:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759341.654379:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12398 00000100:00100000:2.0:1787759341.654379:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655304960/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759341.654386:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759341.654393:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759341.654399:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787759341.654400:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759341.654407:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759341.654412:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655305600/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759341.654414:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655304960:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787759341.654427:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655305088:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787759341.654429:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759341.654435:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787759341.654437:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759341.654442:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12398 00000100:00080000:1.0:1787759341.654447:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759341.654451:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759341.654456:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787759341.654458:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759341.654470:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655305728/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759341.654477:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759341/real 1787759341] req@ffff8c05b2d9b480 x1874588655305088/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759341.654482:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759341.654489:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759341.654492:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759341.654498:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12398 00000100:00080000:1.0:1787759341.654503:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787759341.654503:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655305088/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759341.654512:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759341.654518:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787759341.654531:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787759341.654534:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655305856/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759341.654539:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759341.654547:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655305088:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759341.654579:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655305216:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759341.654600:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759341.654611:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759341/real 1787759341] req@ffff8c05b2d99f80 x1874588655305216/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759341.654623:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655305216/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759341.654631:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759341.654634:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759341.654637:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655305216:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759341.654642:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655305344:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759341.654659:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759341.654668:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759341/real 1787759341] req@ffff8c05b2d9bb80 x1874588655305344/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759341.654679:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655305344/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759341.654687:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759341.654690:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759341.654693:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655305344:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759341.654699:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655305472:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759341.654728:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759341.654740:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759341/real 1787759341] req@ffff8c05b2d9b100 x1874588655305472/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759341.654750:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655305472/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759341.654758:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759341.654761:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759341.654764:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655305472:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759341.654770:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655305600:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759341.654789:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759341.654799:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759341/real 1787759341] req@ffff8c05b2d98000 x1874588655305600/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759341.654809:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655305600/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759341.654816:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759341.654819:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759341.654822:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655305600:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759341.654827:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655305728:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759341.654845:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759341.654856:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759341/real 1787759341] req@ffff8c05b2d98700 x1874588655305728/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759341.654867:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655305728/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759341.654900:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759341.654904:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759341.654908:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655305728:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759341.654914:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655305856:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759341.654933:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759341.654943:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759341/real 1787759341] req@ffff8c05b2d99880 x1874588655305856/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759341.654955:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655305856/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759341.654963:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759341.654966:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759341.654968:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655305856:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759341.781969:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759342.230134:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759342.806053:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759343.254138:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759343.830005:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759344.278093:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759344.854038:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759345.302158:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759345.877977:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759346.326127:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787759346.774043:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759346.774054:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787759346.774058:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759346.774067:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12403 00000100:00080000:0.0:1787759346.774083:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759346.774091:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787759346.774125:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655305984/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759346.774150:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759346.774153:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787759346.774155:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759346.774159:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12403 00000100:00080000:0.0:1787759346.774163:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759346.774166:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759346.774176:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759346.774190:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588655306112/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759346.774201:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759346.774205:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787759346.774208:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787759346.774209:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655305984:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787759346.774214:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12403 00000100:00080000:0.0:1787759346.774221:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759346.774224:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759346.774228:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759346.774238:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588655306240/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759346.774246:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759346.774248:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787759346.774249:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759346.774252:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12403 00000100:00080000:0.0:1787759346.774256:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759346.774258:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759346.774260:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759346.774268:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588655306368/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759346.774274:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787759346.774278:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787759346.774279:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787759346.774282:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759346.774287:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12403 00000100:00080000:0.0:1787759346.774291:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759346.774295:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759346.774300:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759346.774312:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588655306496/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759346.774323:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787759346.774325:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759346/real 1787759346] req@ffff8c05a9f75880 x1874588655305984/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759346.774327:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787759346.774332:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759346.774339:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12403 00000100:00080000:0.0:1787759346.774344:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787759346.774345:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655305984/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759346.774349:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759346.774356:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787759346.774364:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787759346.774369:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588655306624/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759346.774373:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787759346.774381:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787759346.774383:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655305984:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787759346.774393:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787759346.774395:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759346.774400:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12403 00000100:00080000:0.0:1787759346.774403:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759346.774407:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759346.774411:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655306112:192.168.201.107@tcp:38:kworker.0 00000008:00100000:0.0:1787759346.774413:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759346.774431:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588655306752/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787759346.774441:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787759346.774443:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759346.774446:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787759346.774448:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759346.774452:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12403 00000100:00080000:0.0:1787759346.774455:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787759346.774458:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759346/real 1787759346] req@ffff8c05a9f76d80 x1874588655306112/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759346.774461:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759346.774468:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759346.774477:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588655306112/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759346.774482:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655306880/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759346.774497:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759346.774506:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759346.774513:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655306112:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759346.774522:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655306240:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759346.774545:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759346.774555:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759346/real 1787759346] req@ffff8c05a9f77100 x1874588655306240/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759346.774565:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588655306240/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759346.774571:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759346.774574:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759346.774576:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655306240:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759346.774582:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655306368:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759346.774599:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759346.774607:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759346/real 1787759346] req@ffff8c05a9f74e00 x1874588655306368/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759346.774618:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588655306368/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759346.774624:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759346.774627:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759346.774629:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655306368:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759346.774634:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655306496:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759346.774649:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759346.774658:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759346/real 1787759346] req@ffff8c05a9f75180 x1874588655306496/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759346.774669:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588655306496/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759346.774676:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759346.774678:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759346.774681:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655306496:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759346.774686:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655306624:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759346.774703:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759346.774713:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759346/real 1787759346] req@ffff8c05a9f74000 x1874588655306624/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759346.774724:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588655306624/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759346.774732:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759346.774735:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759346.774738:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655306624:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759346.774756:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655306752:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759346.774789:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759346.774798:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759346/real 1787759346] req@ffff8c05a9f75500 x1874588655306752/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759346.774808:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588655306752/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759346.774816:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759346.774819:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759346.774822:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655306752:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759346.774828:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655306880:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759346.774847:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759346.774857:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759346/real 1787759346] req@ffff8c05a9f75880 x1874588655306880/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759346.774867:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655306880/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759346.774902:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759346.774905:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759346.774907:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655306880:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759346.902010:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759347.350203:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759347.926015:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759348.374121:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759348.949995:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759349.398132:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759349.974022:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759350.422107:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759350.997963:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759351.446053:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787759351.894015:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759351.894026:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787759351.894030:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759351.894041:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12408 00000100:00080000:0.0:1787759351.894052:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759351.894059:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787759351.894095:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588655307008/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759351.894119:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759351.894122:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787759351.894124:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759351.894128:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12408 00000100:00080000:0.0:1787759351.894132:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759351.894136:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759351.894144:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759351.894148:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655307008:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787759351.894159:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655307136/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759351.894169:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759351.894172:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787759351.894174:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759351.894178:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12408 00000100:00080000:0.0:1787759351.894181:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759351.894186:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759351.894190:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759351.894202:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588655307264/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787759351.894208:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787759351.894216:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759351.894220:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787759351.894222:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759351.894227:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12408 00000100:00080000:0.0:1787759351.894232:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759351.894236:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759351.894242:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759351.894255:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759351/real 1787759351] req@ffff8c05a9f75c00 x1874588655307008/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759351.894257:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655307392/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759351.894277:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759351.894282:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1787759351.894282:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588655307008/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759351.894286:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759351.894294:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12408 00000100:00080000:2.0:1787759351.894298:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759351.894299:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759351.894303:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787759351.894307:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759351.894308:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655307008:192.168.201.107@tcp:250:kworker.0 00000008:00100000:0.0:1787759351.894316:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759351.894327:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588655307520/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759351.894332:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655307136:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787759351.894339:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759351.894343:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787759351.894344:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759351.894347:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12408 00000100:00080000:0.0:1787759351.894350:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759351.894353:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:2.0:1787759351.894356:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:0.0:1787759351.894357:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759351.894368:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655307648/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759351.894370:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759351/real 1787759351] req@ffff8c05a9f76300 x1874588655307136/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759351.894382:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759351.894389:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787759351.894390:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759351.894394:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12408 00000100:00080000:0.0:1787759351.894395:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787759351.894398:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655307136/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759351.894399:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759351.894404:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759351.894413:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655307776/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759351.894414:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759351.894423:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787759351.894424:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787759351.894428:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655307136:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787759351.894429:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787759351.894432:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759351.894438:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12408 00000100:00080000:0.0:1787759351.894440:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787759351.894441:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655307264:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787759351.894445:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759351.894451:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759351.894459:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655307904/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787759351.894476:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759351.894490:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759351/real 1787759351] req@ffff8c05a9f76a00 x1874588655307264/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759351.894499:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588655307264/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759351.894504:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759351.894506:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759351.894508:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655307264:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759351.894513:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655307392:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759351.894526:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759351.894532:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759351/real 1787759351] req@ffff8c05a9f77800 x1874588655307392/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759351.894540:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655307392/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759351.894545:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759351.894547:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759351.894550:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655307392:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759351.894562:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655307520:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759351.894574:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759351.894583:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759351/real 1787759351] req@ffff8c05a9f77b80 x1874588655307520/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759351.894591:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588655307520/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759351.894598:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759351.894601:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759351.894604:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655307520:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759351.894607:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655307648:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759351.894619:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759351.894626:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759351/real 1787759351] req@ffff8c05a9f76680 x1874588655307648/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759351.894635:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655307648/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759351.894640:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759351.894642:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759351.894643:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655307648:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759351.894646:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655307776:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759351.894654:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759351.894658:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759351/real 1787759351] req@ffff8c05a9f74380 x1874588655307776/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759351.894664:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655307776/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759351.894674:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759351.894675:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759351.894677:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655307776:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759351.894680:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655307904:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759351.894688:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759351.894692:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759351/real 1787759351] req@ffff8c05a9f74a80 x1874588655307904/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759351.894698:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655307904/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759351.894702:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759351.894704:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759351.894705:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655307904:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759352.022020:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759352.470083:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759353.046025:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759353.494065:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759354.070116:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759354.518115:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759355.093996:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759355.542096:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759356.117991:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759356.566097:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787759357.014015:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759357.014028:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787759357.014033:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759357.014044:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12413 00000100:00080000:1.0:1787759357.014054:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759357.014062:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759357.014109:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655308032/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759357.014133:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759357.014136:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759357.014137:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759357.014139:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12413 00000100:00080000:1.0:1787759357.014142:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759357.014144:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759357.014150:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759357.014168:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588655308160/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759357.014186:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655308032:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787759357.014187:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759357.014195:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759357.014198:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759357.014202:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12413 00000100:00080000:1.0:1787759357.014206:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759357.014208:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759357.014213:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759357.014222:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655308288/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759357.014229:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759357.014231:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759357.014232:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759357.014235:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12413 00000100:00080000:1.0:1787759357.014237:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759357.014239:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759357.014242:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759357.014248:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655308416/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759357.014253:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759357.014255:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759357.014256:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759357.014260:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12413 00000800:00000100:2.0:1787759357.014263:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787759357.014265:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759357.014269:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759357.014272:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759357.014278:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655308544/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759357.014283:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759357.014286:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759357.014287:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759357.014290:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12413 00000100:00080000:1.0:1787759357.014291:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759357.014293:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759357.014295:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759357.014301:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655308672/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759357.014307:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759357.014309:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787759357.014310:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787759357.014314:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759357/real 1787759357] req@ffff8c05b2d98380 x1874588655308032/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759357.014316:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12413 00000100:00080000:1.0:1787759357.014320:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759357.014326:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759357.014331:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759357.014337:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655308032/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759357.014341:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655308800/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759357.014359:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759357.014365:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759357.014367:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787759357.014370:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759357.014373:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787759357.014374:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655308032:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787759357.014378:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12413 00000100:00080000:1.0:1787759357.014382:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759357.014384:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759357.014387:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759357.014393:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655308928/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759357.014399:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655308160:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759357.014418:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759357.014427:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759357/real 1787759357] req@ffff8c05b2d9a300 x1874588655308160/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759357.014434:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588655308160/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759357.014440:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759357.014442:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759357.014444:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655308160:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759357.014447:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655308288:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759357.014456:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759357.014461:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759357/real 1787759357] req@ffff8c05b2d99880 x1874588655308288/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759357.014468:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655308288/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759357.014473:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759357.014474:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759357.014476:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655308288:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759357.014479:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655308416:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759357.014489:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759357.014493:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759357/real 1787759357] req@ffff8c05b2d98700 x1874588655308416/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759357.014500:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655308416/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759357.014504:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759357.014506:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759357.014507:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655308416:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759357.014510:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655308544:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759357.014519:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759357.014525:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759357/real 1787759357] req@ffff8c05b2d98000 x1874588655308544/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759357.014536:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655308544/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759357.014543:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759357.014546:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759357.014548:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655308544:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759357.014554:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655308672:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759357.014573:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759357.014582:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759357/real 1787759357] req@ffff8c05b2d9b100 x1874588655308672/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759357.014589:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655308672/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759357.014597:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759357.014599:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759357.014601:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655308672:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759357.014604:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655308800:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759357.014616:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759357.014621:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759357/real 1787759357] req@ffff8c05b2d9bb80 x1874588655308800/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759357.014627:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655308800/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759357.014632:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759357.014633:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759357.014635:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655308800:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759357.014647:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655308928:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759357.014656:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759357.014662:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759357/real 1787759357] req@ffff8c05b2d99f80 x1874588655308928/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759357.014668:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655308928/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759357.014672:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759357.014674:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759357.014676:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655308928:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759357.141972:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759357.590135:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759358.166013:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759358.614069:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759359.190011:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759359.638129:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759360.214028:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759360.662088:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759361.238013:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759361.686178:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787759362.134028:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759362.134037:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787759362.134040:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759362.134050:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12418 00000100:00080000:2.0:1787759362.134060:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759362.134065:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759362.134097:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588655309056/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759362.134119:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759362.134122:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759362.134124:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759362.134128:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12418 00000100:00080000:2.0:1787759362.134132:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759362.134136:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759362.134145:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759362.134152:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655309056:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787759362.134163:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588655309184/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759362.134172:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759362.134175:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759362.134177:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759362.134180:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12418 00000100:00080000:2.0:1787759362.134183:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759362.134187:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759362.134190:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759362.134199:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588655309312/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759362.134206:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787759362.134209:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787759362.134209:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759362.134211:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759362.134214:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12418 00000100:00080000:2.0:1787759362.134216:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759362.134219:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759362.134223:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759362.134231:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588655309440/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759362.134238:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759362.134241:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759362.134242:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759362.134245:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12418 00000100:00080000:2.0:1787759362.134247:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759362.134250:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759362.134254:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759362.134262:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588655309568/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759362.134269:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759362.134272:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759362.134273:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759362.134276:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12418 00000100:00080000:2.0:1787759362.134279:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787759362.134281:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759362/real 1787759362] req@ffff8c05b2e72a00 x1874588655309056/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759362.134282:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759362.134285:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759362.134293:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588655309696/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759362.134297:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588655309056/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759362.134301:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759362.134304:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787759362.134305:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759362.134306:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759362.134309:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12418 00000100:00080000:1.0:1787759362.134310:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787759362.134312:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787759362.134313:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655309056:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787759362.134315:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759362.134318:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759362.134326:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588655309824/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759362.134333:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759362.134336:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759362.134337:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759362.134339:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12418 00000100:00080000:2.0:1787759362.134342:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759362.134345:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759362.134347:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655309184:192.168.201.107@tcp:38:kworker.0 00000008:00100000:2.0:1787759362.134349:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759362.134357:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588655309952/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787759362.134367:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759362.134377:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759362/real 1787759362] req@ffff8c05b2e70700 x1874588655309184/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759362.134387:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588655309184/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759362.134395:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759362.134397:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759362.134400:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655309184:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759362.134406:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655309312:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759362.134424:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759362.134431:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759362/real 1787759362] req@ffff8c05b2e73480 x1874588655309312/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759362.134438:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588655309312/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759362.134443:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759362.134445:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759362.134447:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655309312:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759362.134450:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655309440:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759362.134460:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759362.134465:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759362/real 1787759362] req@ffff8c05b2e71500 x1874588655309440/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759362.134475:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588655309440/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759362.134480:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759362.134482:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759362.134485:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655309440:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759362.134490:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655309568:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759362.134506:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759362.134516:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759362/real 1787759362] req@ffff8c05b2e73b80 x1874588655309568/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759362.134527:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588655309568/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759362.134534:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759362.134536:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759362.134538:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655309568:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759362.134541:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655309696:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759362.134553:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759362.134560:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759362/real 1787759362] req@ffff8c05b2e71c00 x1874588655309696/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759362.134566:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588655309696/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759362.134574:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759362.134576:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759362.134578:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655309696:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759362.134580:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655309824:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759362.134589:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759362.134593:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759362/real 1787759362] req@ffff8c05b2e71880 x1874588655309824/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759362.134599:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588655309824/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759362.134604:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759362.134605:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759362.134607:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655309824:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759362.134618:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655309952:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759362.134627:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759362.134634:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759362/real 1787759362] req@ffff8c05b2e70e00 x1874588655309952/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759362.134644:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588655309952/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759362.134651:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759362.134654:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759362.134658:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655309952:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759362.261989:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759362.710133:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759363.285970:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759363.734206:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759364.310010:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759364.758077:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759365.333990:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759365.782198:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759366.357997:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759366.806119:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787759367.254004:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759367.254022:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787759367.254026:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759367.254037:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12423 00000100:00080000:2.0:1787759367.254049:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759367.254056:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759367.254105:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73800 x1874588655310080/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759367.254131:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759367.254134:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759367.254136:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759367.254141:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12423 00000100:00080000:2.0:1787759367.254145:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759367.254148:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759367.254157:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759367.254174:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588655310208/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759367.254181:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655310080:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787759367.254186:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759367.254191:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759367.254192:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759367.254197:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12423 00000100:00080000:2.0:1787759367.254202:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759367.254205:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759367.254211:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759367.254223:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588655310336/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759367.254232:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759367.254235:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759367.254237:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759367.254242:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12423 00000800:00000100:1.0:1787759367.254243:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787759367.254247:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759367.254251:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759367.254256:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759367.254268:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588655310464/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759367.254275:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759367.254278:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759367.254279:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759367.254281:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12423 00000100:00080000:2.0:1787759367.254284:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759367.254287:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759367.254290:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759367.254304:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759367/real 1787759367] req@ffff8c05b2e73800 x1874588655310080/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759367.254305:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588655310592/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759367.254324:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759367.254326:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759367.254327:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759367.254332:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12423 00000100:00100000:1.0:1787759367.254341:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73800 x1874588655310080/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759367.254342:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759367.254349:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759367.254356:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787759367.254359:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759367.254366:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759367.254370:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588655310720/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787759367.254373:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655310080:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787759367.254387:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759367.254390:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787759367.254392:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759367.254396:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12423 00000100:00080000:2.0:1787759367.254400:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787759367.254405:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655310208:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787759367.254405:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759367.254412:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759367.254426:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588655310848/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759367.254436:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787759367.254440:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787759367.254442:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759367.254445:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759367.254450:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12423 00000100:00080000:2.0:1787759367.254455:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787759367.254458:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759367/real 1787759367] req@ffff8c05b2e70e00 x1874588655310208/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759367.254462:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759367.254470:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759367.254480:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588655310208/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759367.254486:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588655310976/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759367.254496:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759367.254503:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759367.254507:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655310208:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759367.254514:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655310336:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759367.254537:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759367.254551:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759367/real 1787759367] req@ffff8c05b2e71880 x1874588655310336/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759367.254561:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588655310336/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759367.254568:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759367.254572:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759367.254574:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655310336:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759367.254579:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655310464:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759367.254598:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759367.254608:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759367/real 1787759367] req@ffff8c05b2e71c00 x1874588655310464/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759367.254618:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588655310464/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759367.254641:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759367.254645:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759367.254647:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655310464:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759367.254653:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655310592:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759367.254674:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759367.254683:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759367/real 1787759367] req@ffff8c05b2e73b80 x1874588655310592/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759367.254697:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588655310592/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759367.254703:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759367.254706:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759367.254708:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655310592:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759367.254713:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655310720:192.168.201.107@tcp:38:kworker.0 00000800:00000100:1.0:1787759367.254730:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759367.254740:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759367/real 1787759367] req@ffff8c05b2e71500 x1874588655310720/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759367.254748:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588655310720/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759367.254755:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759367.254758:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759367.254761:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655310720:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759367.254780:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655310848:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759367.254799:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759367.254809:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759367/real 1787759367] req@ffff8c05b2e73480 x1874588655310848/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759367.254816:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588655310848/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759367.254825:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759367.254826:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759367.254828:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655310848:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759367.254831:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655310976:192.168.201.107@tcp:8:kworker.0 00000800:00000100:1.0:1787759367.254846:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759367.254855:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759367/real 1787759367] req@ffff8c05b2e70700 x1874588655310976/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759367.254865:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588655310976/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759367.254872:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759367.254911:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759367.254924:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655310976:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759367.382030:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759367.830179:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759368.406002:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759368.854309:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759369.430059:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759369.878079:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759370.453981:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759370.902238:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759371.477964:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759371.926239:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787759372.374057:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759372.374070:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787759372.374074:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759372.374086:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12428 00000100:00080000:1.0:1787759372.374100:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759372.374109:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759372.374150:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655311104/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759372.374186:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759372.374189:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759372.374191:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759372.374196:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12428 00000100:00080000:1.0:1787759372.374201:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759372.374206:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759372.374216:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759372.374216:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655311104:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787759372.374236:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655311232/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759372.374247:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759372.374251:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759372.374253:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759372.374257:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12428 00000100:00080000:1.0:1787759372.374261:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759372.374265:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759372.374270:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759372.374284:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655311360/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787759372.374291:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787759372.374299:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759372.374302:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759372.374305:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759372.374310:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12428 00000100:00080000:1.0:1787759372.374317:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759372.374321:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759372.374326:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759372.374339:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655311488/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759372.374345:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759372/real 1787759372] req@ffff8c05b2d9b480 x1874588655311104/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759372.374355:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759372.374360:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759372.374362:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787759372.374366:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655311104/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759372.374368:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12428 00000100:00080000:1.0:1787759372.374375:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759372.374379:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759372.374383:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:2.0:1787759372.374386:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759372.374391:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655311104:192.168.201.107@tcp:250:kworker.0 00000008:00100000:1.0:1787759372.374392:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759372.374407:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655311616/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759372.374414:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655311232:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787759372.374426:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759372.374429:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759372.374432:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759372.374438:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12428 00000800:00000100:2.0:1787759372.374439:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787759372.374445:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787759372.374453:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759372/real 1787759372] req@ffff8c05b2d9aa00 x1874588655311232/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759372.374465:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759372.374465:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655311232/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787759372.374475:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787759372.374476:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759372.374481:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759372.374485:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655311232:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759372.374490:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655311360:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759372.374494:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655311744/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759372.374512:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787759372.374516:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787759372.374519:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787759372.374523:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787759372.374527:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759372/real 1787759372] req@ffff8c05b2d99f80 x1874588655311360/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759372.374530:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12428 00000100:00080000:1.0:1787759372.374549:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787759372.374554:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655311360/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759372.374558:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:2.0:1787759372.374566:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787759372.374568:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787759372.374570:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759372.374575:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655311360:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759372.374581:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655311488:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759372.374586:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655311872/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787759372.374597:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787759372.374598:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759372.374604:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759372.374606:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787759372.374608:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759372/real 1787759372] req@ffff8c05b2d9bb80 x1874588655311488/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759372.374613:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12428 00000100:00080000:1.0:1787759372.374622:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787759372.374625:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655311488/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759372.374630:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:2.0:1787759372.374634:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787759372.374638:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787759372.374639:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759372.374643:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655311488:192.168.201.107@tcp:8:kworker.0 00000100:00100000:1.0:1787759372.374658:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655312000/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759372.374659:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655311616:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759372.374677:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759372.374687:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759372/real 1787759372] req@ffff8c05b2d9b100 x1874588655311616/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759372.374694:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655311616/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759372.374699:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759372.374701:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759372.374703:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655311616:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759372.374708:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655311744:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759372.374717:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759372.374722:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759372/real 1787759372] req@ffff8c05b2d98000 x1874588655311744/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759372.374729:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655311744/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759372.374733:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759372.374734:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759372.374736:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655311744:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759372.374739:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655311872:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759372.374747:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759372.374752:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759372/real 1787759372] req@ffff8c05b2d98700 x1874588655311872/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759372.374758:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655311872/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759372.374763:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759372.374765:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759372.374767:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655311872:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759372.374778:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655312000:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759372.374796:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759372.374806:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759372/real 1787759372] req@ffff8c05b2d99880 x1874588655312000/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759372.374817:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655312000/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759372.374825:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759372.374827:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759372.374830:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655312000:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759372.502017:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759372.950085:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759373.526032:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759373.974293:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759374.549973:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759374.998190:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759375.574025:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759376.022167:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759376.597974:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759377.046043:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787759377.493990:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759377.494001:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787759377.494005:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759377.494016:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12433 00000100:00080000:1.0:1787759377.494027:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759377.494034:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759377.494069:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588655312128/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759377.494102:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759377.494105:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759377.494107:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759377.494111:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12433 00000100:00080000:1.0:1787759377.494114:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759377.494118:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759377.494127:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759377.494141:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655312256/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759377.494153:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655312128:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787759377.494154:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759377.494162:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759377.494164:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759377.494169:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12433 00000100:00080000:1.0:1787759377.494171:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759377.494174:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759377.494179:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759377.494191:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655312384/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759377.494206:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759377.494210:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759377.494212:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759377.494216:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12433 00000800:00000100:2.0:1787759377.494219:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787759377.494222:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759377.494228:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759377.494233:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759377.494246:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655312512/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759377.494255:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787759377.494259:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759377/real 1787759377] req@ffff8c05b2d9a300 x1874588655312128/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759377.494260:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759377.494264:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759377.494272:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12433 00000100:00080000:1.0:1787759377.494277:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787759377.494278:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588655312128/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759377.494285:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:2.0:1787759377.494291:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787759377.494293:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787759377.494297:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759377.494301:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655312128:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787759377.494308:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655312640/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759377.494324:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787759377.494326:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655312256:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787759377.494330:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759377.494334:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759377.494339:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12433 00000100:00080000:1.0:1787759377.494343:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759377.494347:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759377.494352:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787759377.494357:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759377.494366:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655312768/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759377.494373:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759377/real 1787759377] req@ffff8c05b2d98380 x1874588655312256/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759377.494382:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759377.494387:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787759377.494390:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655312256/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759377.494391:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759377.494400:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12433 00000100:00080000:2.0:1787759377.494402:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759377.494404:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759377.494408:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787759377.494413:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759377.494413:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655312256:192.168.201.107@tcp:38:kworker.0 00000008:00100000:1.0:1787759377.494423:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759377.494423:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655312384:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759377.494440:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655312896/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759377.494462:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759377.494467:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:2.0:1787759377.494468:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787759377.494470:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759377.494476:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12433 00000100:00080000:1.0:1787759377.494481:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787759377.494485:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759377/real 1787759377] req@ffff8c05b2d99880 x1874588655312384/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759377.494488:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759377.494496:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759377.494508:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655312384/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759377.494513:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655313024/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759377.494530:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759377.494539:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759377.494547:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655312384:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759377.494556:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655312512:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759377.494581:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759377.494592:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759377/real 1787759377] req@ffff8c05b2d98700 x1874588655312512/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759377.494604:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655312512/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759377.494612:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759377.494615:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759377.494619:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655312512:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759377.494633:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655312640:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759377.494652:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759377.494662:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759377/real 1787759377] req@ffff8c05b2d98000 x1874588655312640/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759377.494673:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655312640/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759377.494681:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759377.494684:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759377.494687:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655312640:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759377.494692:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655312768:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759377.494710:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759377.494719:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759377/real 1787759377] req@ffff8c05b2d9b100 x1874588655312768/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759377.494731:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655312768/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759377.494744:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759377.494747:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759377.494750:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655312768:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759377.494755:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655312896:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759377.494771:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759377.494779:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759377/real 1787759377] req@ffff8c05b2d9bb80 x1874588655312896/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759377.494789:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655312896/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759377.494797:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759377.494800:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759377.494803:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655312896:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759377.494808:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655313024:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759377.494825:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759377.494835:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759377/real 1787759377] req@ffff8c05b2d99f80 x1874588655313024/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759377.494845:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655313024/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759377.494854:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759377.494857:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759377.494861:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655313024:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759377.621989:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759378.070243:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759378.645977:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759379.094471:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759379.670153:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759380.118257:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759380.694138:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759381.142105:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759381.717961:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759382.166164:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787759382.613990:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759382.614000:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787759382.614004:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759382.614012:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12439 00000100:00080000:1.0:1787759382.614023:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759382.614028:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759382.614068:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655313152/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759382.614097:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759382.614100:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759382.614102:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759382.614106:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12439 00000100:00080000:1.0:1787759382.614109:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759382.614112:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759382.614122:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759382.614148:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655313280/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759382.614148:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655313152:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787759382.614166:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759382.614169:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759382.614170:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759382.614174:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12439 00000100:00080000:1.0:1787759382.614178:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759382.614181:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759382.614186:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759382.614200:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655313408/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759382.614207:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759382.614210:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759382.614211:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759382.614214:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12439 00000100:00080000:1.0:1787759382.614216:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759382.614221:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:2.0:1787759382.614223:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:1.0:1787759382.614227:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759382.614241:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655313536/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759382.614250:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759382.614253:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759382.614255:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759382.614259:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12439 00000100:00080000:1.0:1787759382.614264:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787759382.614265:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759382/real 1787759382] req@ffff8c05b2d9aa00 x1874588655313152/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759382.614271:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759382.614279:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759382.614285:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655313152/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759382.614291:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655313664/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759382.614296:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759382.614303:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787759382.614311:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787759382.614311:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655313152:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787759382.614320:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759382.614322:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759382.614325:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12439 00000100:00080000:1.0:1787759382.614328:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759382.614330:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759382.614333:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759382.614339:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655313792/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759382.614344:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655313280:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787759382.614348:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759382.614352:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787759382.614353:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759382.614371:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12439 00000100:00080000:1.0:1787759382.614375:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759382.614378:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759382.614381:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787759382.614389:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759382.614393:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655313920/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759382.614400:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759382.614402:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759382.614404:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759382.614408:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12439 00000100:00100000:2.0:1787759382.614409:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759382/real 1787759382] req@ffff8c05b2d9b480 x1874588655313280/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759382.614411:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759382.614417:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759382.614423:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759382.614431:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655313280/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759382.614433:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655314048/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759382.614459:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759382.614466:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759382.614470:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655313280:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759382.614477:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655313408:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759382.614501:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759382.614513:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759382/real 1787759382] req@ffff8c05b2d99f80 x1874588655313408/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759382.614525:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655313408/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759382.614532:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759382.614535:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759382.614539:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655313408:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759382.614544:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655313536:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759382.614562:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759382.614570:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759382/real 1787759382] req@ffff8c05b2d9bb80 x1874588655313536/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759382.614577:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655313536/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759382.614582:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759382.614584:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759382.614586:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655313536:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759382.614590:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655313664:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759382.614600:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759382.614605:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759382/real 1787759382] req@ffff8c05b2d9b100 x1874588655313664/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759382.614611:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655313664/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759382.614616:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759382.614618:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759382.614620:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655313664:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759382.614632:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655313792:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759382.614641:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759382.614648:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759382/real 1787759382] req@ffff8c05b2d98000 x1874588655313792/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759382.614658:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655313792/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759382.614665:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759382.614668:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759382.614671:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655313792:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759382.614676:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655313920:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759382.614694:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759382.614702:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759382/real 1787759382] req@ffff8c05b2d98700 x1874588655313920/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759382.614709:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655313920/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759382.614714:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759382.614716:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759382.614718:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655313920:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759382.614721:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655314048:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759382.614740:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759382.614746:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759382/real 1787759382] req@ffff8c05b2d99880 x1874588655314048/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759382.614752:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655314048/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759382.614756:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759382.614758:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759382.614760:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655314048:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759382.742004:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759383.190154:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759383.765967:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759384.214166:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759384.790073:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759385.238042:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759385.814056:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759386.262211:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759386.837993:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759387.287362:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787759387.734023:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759387.734035:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787759387.734039:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759387.734051:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12444 00000100:00080000:1.0:1787759387.734062:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759387.734072:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759387.734111:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655314176/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759387.734148:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759387.734152:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759387.734153:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759387.734158:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12444 00000100:00080000:1.0:1787759387.734161:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759387.734166:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759387.734175:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759387.734192:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588655314304/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759387.734197:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655314176:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787759387.734218:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759387.734222:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759387.734225:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759387.734231:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12444 00000100:00080000:1.0:1787759387.734249:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759387.734254:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:0.0:1787759387.734258:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:1.0:1787759387.734262:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759387.734293:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655314432/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759387.734308:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759387.734312:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759387.734314:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787759387.734318:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759387/real 1787759387] req@ffff8c05b2d98380 x1874588655314176/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759387.734322:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12444 00000100:00080000:1.0:1787759387.734328:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759387.734335:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759387.734342:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759387.734347:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655314176/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759387.734360:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655314560/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759387.734361:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759387.734371:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787759387.734380:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787759387.734382:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655314176:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787759387.734387:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759387.734393:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759387.734400:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12444 00000100:00080000:1.0:1787759387.734411:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759387.734416:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787759387.734422:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655314304:192.168.201.107@tcp:38:kworker.0 00000008:00100000:1.0:1787759387.734423:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759387.734442:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655314688/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759387.734455:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759387.734459:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759387.734461:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759387.734466:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12444 00000100:00080000:1.0:1787759387.734470:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759387.734474:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759387.734480:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759387.734489:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655314816/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759387.734498:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759387.734501:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787759387.734503:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759387.734507:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12444 00000100:00080000:1.0:1787759387.734511:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787759387.734514:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787759387.734518:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759387.734524:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759387.734537:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759387/real 1787759387] req@ffff8c05b2d9a300 x1874588655314304/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759387.734538:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655314944/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759387.734574:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588655314304/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759387.734575:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759387.734583:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759387.734587:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759387.734593:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759387.734594:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12444 00000100:00080000:1.0:1787759387.734599:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759387.734603:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787759387.734608:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787759387.734611:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655314304:192.168.201.107@tcp:38:kworker.0 00000008:00100000:1.0:1787759387.734615:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759387.734621:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655314432:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759387.734633:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655315072/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787759387.734655:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759387.734690:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759387/real 1787759387] req@ffff8c05b2d99880 x1874588655314432/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759387.734706:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655314432/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759387.734714:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759387.734717:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759387.734720:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655314432:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759387.734726:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655314560:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759387.734743:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759387.734752:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759387/real 1787759387] req@ffff8c05b2d98700 x1874588655314560/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759387.734761:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655314560/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759387.734767:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759387.734770:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759387.734774:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655314560:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759387.734937:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655314688:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759387.734962:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759387.734972:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759387/real 1787759387] req@ffff8c05b2d98000 x1874588655314688/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759387.734979:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655314688/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759387.734986:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759387.734989:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759387.734992:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655314688:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759387.734997:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655314816:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759387.735015:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759387.735025:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759387/real 1787759387] req@ffff8c05b2d9b100 x1874588655314816/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759387.735036:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655314816/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759387.735041:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759387.735043:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759387.735045:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655314816:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759387.735048:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655314944:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759387.735060:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759387.735066:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759387/real 1787759387] req@ffff8c05b2d9bb80 x1874588655314944/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759387.735077:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655314944/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759387.735082:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759387.735083:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759387.735085:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655314944:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759387.735088:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655315072:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759387.735096:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759387.735101:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759387/real 1787759387] req@ffff8c05b2d99f80 x1874588655315072/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759387.735107:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655315072/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759387.735112:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759387.735113:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759387.735115:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655315072:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759387.861963:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759388.310085:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759388.886002:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759389.334104:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759389.913615:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759390.359273:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759390.934018:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759391.382073:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759391.958007:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759392.406161:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787759392.854116:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759392.854128:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787759392.854132:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759392.854143:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12449 00000100:00080000:2.0:1787759392.854155:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759392.854162:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759392.854231:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588655315200/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759392.854257:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759392.854260:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759392.854262:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759392.854266:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12449 00000100:00080000:2.0:1787759392.854269:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759392.854274:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759392.854282:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759392.854294:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588655315328/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759392.854303:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759392.854306:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759392.854308:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759392.854311:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12449 00000100:00080000:2.0:1787759392.854313:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759392.854317:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759392.854321:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759392.854330:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588655315456/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759392.854338:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759392.854341:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759392.854342:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759392.854346:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12449 00000100:00080000:2.0:1787759392.854348:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759392.854352:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759392.854356:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759392.854367:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588655315584/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759392.854375:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759392.854378:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759392.854380:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759392.854383:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12449 00000100:00080000:2.0:1787759392.854386:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759392.854389:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759392.854393:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759392.854402:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588655315712/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759392.854410:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759392.854413:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759392.854414:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759392.854417:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12449 00000100:00080000:2.0:1787759392.854420:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759392.854423:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759392.854427:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759392.854435:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588655315840/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759392.854442:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759392.854445:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787759392.854446:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759392.854449:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12449 00000100:00080000:2.0:1787759392.854452:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759392.854454:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759392.854459:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759392.854469:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588655315968/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759392.854477:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759392.854480:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759392.854481:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759392.854484:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12449 00000100:00080000:2.0:1787759392.854487:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759392.854491:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759392.854495:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759392.854504:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588655316096/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759392.854894:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655315200:192.168.201.107@tcp:250:kworker.0 00000800:00000100:0.0:1787759392.854956:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759392.856053:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759392/real 1787759392] req@ffff8c05b2e72a00 x1874588655315200/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759392.856071:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588655315200/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759392.856081:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759392.856084:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759392.856088:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655315200:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787759392.856094:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655315328:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759392.856116:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759392.856127:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759392/real 1787759392] req@ffff8c05b2e72d80 x1874588655315328/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759392.856137:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588655315328/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759392.856144:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759392.856146:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759392.856149:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655315328:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759392.856154:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655315456:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759392.856188:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759392.856198:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759392/real 1787759392] req@ffff8c05b2e71180 x1874588655315456/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759392.856216:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588655315456/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759392.856224:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759392.856226:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759392.856229:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655315456:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759392.856235:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655315584:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759392.856250:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759392.856258:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759392/real 1787759392] req@ffff8c05b2e72680 x1874588655315584/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759392.856267:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588655315584/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759392.856273:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759392.856276:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759392.856278:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655315584:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759392.856282:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655315712:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759392.856295:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759392.856302:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759392/real 1787759392] req@ffff8c05b2e70380 x1874588655315712/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759392.856313:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588655315712/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759392.856318:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759392.856320:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759392.856322:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655315712:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759392.856326:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655315840:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759392.856338:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759392.856345:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759392/real 1787759392] req@ffff8c05b2e70000 x1874588655315840/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759392.856356:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588655315840/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759392.856363:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759392.856365:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759392.856368:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655315840:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759392.856372:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655315968:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759392.856386:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759392.856395:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759392/real 1787759392] req@ffff8c05b2e70a80 x1874588655315968/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759392.856404:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588655315968/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759392.856410:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759392.856413:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759392.856415:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655315968:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759392.856420:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655316096:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759392.856431:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759392.856438:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759392/real 1787759392] req@ffff8c05b2e72300 x1874588655316096/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759392.856447:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588655316096/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759392.856459:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759392.856462:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759392.856464:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655316096:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759392.982037:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759393.430281:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759394.006233:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759394.454192:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759395.030042:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759395.478122:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759396.054018:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759396.502154:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759397.078069:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759397.526389:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787759397.974227:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759397.974240:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787759397.974244:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759397.974254:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12454 00000100:00080000:1.0:1787759397.974265:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759397.974273:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759397.974314:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655316224/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759397.974341:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759397.974344:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759397.974347:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759397.974350:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12454 00000100:00080000:1.0:1787759397.974354:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759397.974358:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759397.974368:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759397.974383:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655316352/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759397.974393:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759397.974396:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759397.974398:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759397.974402:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12454 00000100:00080000:1.0:1787759397.974405:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787759397.974411:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655316224:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787759397.974411:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759397.974418:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759397.974431:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655316480/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759397.974442:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759397.974445:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759397.974447:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759397.974451:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12454 00000100:00080000:1.0:1787759397.974470:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759397.974475:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759397.974479:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759397.974491:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655316608/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759397.974500:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759397.974504:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759397.974506:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759397.974509:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12454 00000100:00080000:1.0:1787759397.974513:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759397.974518:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759397.974522:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759397.974534:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655316352:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759397.974534:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655316736/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759397.974548:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759397.974551:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759397.974553:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759397.974557:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12454 00000100:00080000:1.0:1787759397.974565:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759397.974569:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759397.974574:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759397.974585:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655316480:192.168.201.107@tcp:38:kworker.0 00000100:00100000:1.0:1787759397.974587:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655316864/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759397.974600:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759397.974603:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787759397.974605:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759397.974610:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12454 00000100:00080000:1.0:1787759397.974614:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787759397.974616:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655316608:192.168.201.107@tcp:8:kworker.0 00000100:00080000:1.0:1787759397.974621:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759397.974627:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759397.974640:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655316992/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759397.974646:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655316736:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787759397.974657:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759397.974661:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759397.974664:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759397.974669:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12454 00000100:00080000:1.0:1787759397.974673:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759397.974677:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787759397.974680:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655316864:192.168.201.107@tcp:38:kworker.0 00000008:00100000:1.0:1787759397.974683:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759397.974699:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655317120/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759397.974712:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655316992:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759397.974738:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655317120:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759398.102022:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759398.550065:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759399.126002:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759399.574437:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759400.150015:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759400.598202:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787759401.046281:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.107:988 00000400:00000100:2.0:1787759401.046353:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.107@tcp at host 192.168.201.107 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787759401.046370:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787759401.046419:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787759401.046428:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787759401.046436:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787759401.046443:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787759401.046464:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787759401.046472:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:2.0:1787759401.046479:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:0.0:1787759401.047119:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759397/real 1787759401] req@ffff8c05b2d99880 x1874588655317120/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759401.047147:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655317120/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759401.047160:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759401.047164:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759401.047169:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655317120:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759401.047176:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759397/real 1787759401] req@ffff8c05b2d98000 x1874588655316864/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759401.047188:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655316864/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759401.047195:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759401.047198:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759401.047200:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655316864:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759401.047206:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759397/real 1787759401] req@ffff8c05b2d98700 x1874588655316992/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759401.047217:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655316992/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759401.047225:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759401.047229:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759401.047231:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655316992:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759401.047237:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759397/real 1787759401] req@ffff8c05b2d99f80 x1874588655316480/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759401.047247:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655316480/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759401.047254:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759401.047258:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759401.047261:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655316480:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759401.047267:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759397/real 1787759401] req@ffff8c05b2d9bb80 x1874588655316608/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759401.047277:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655316608/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759401.047284:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759401.047287:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759401.047290:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655316608:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759401.047296:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759397/real 1787759401] req@ffff8c05b2d9b100 x1874588655316736/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759401.047306:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655316736/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759401.047314:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759401.047317:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759401.047320:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655316736:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759401.047326:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759397/real 1787759401] req@ffff8c05b2d9b480 x1874588655316224/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759401.047337:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655316224/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759401.047344:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759401.047347:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759401.047351:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655316224:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787759401.047357:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759397/real 1787759401] req@ffff8c05b2d9aa00 x1874588655316352/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759401.047367:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655316352/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759401.047374:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759401.047377:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759401.047379:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655316352:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759401.173976:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759401.623513:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759402.200705:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759402.647582:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787759403.094164:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759403.094177:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787759403.094181:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759403.094191:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12459 00000100:00080000:1.0:1787759403.094203:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759403.094210:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759403.094244:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588655317248/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759403.094276:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759403.094279:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759403.094281:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759403.094285:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12459 00000100:00080000:1.0:1787759403.094288:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759403.094291:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759403.094299:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759403.094310:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655317376/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759403.094318:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759403.094322:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759403.094323:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759403.094326:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12459 00000100:00080000:1.0:1787759403.094329:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759403.094332:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759403.094335:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759403.094343:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655317504/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759403.094351:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759403.094354:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759403.094356:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759403.094360:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12459 00000100:00080000:1.0:1787759403.094362:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759403.094365:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759403.094370:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759403.094377:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655317632/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759403.094385:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759403.094388:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759403.094389:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759403.094392:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12459 00000100:00080000:1.0:1787759403.094395:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759403.094397:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759403.094401:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759403.094407:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655317760/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759403.094634:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759403.094637:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759403.094639:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759403.094643:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12459 00000100:00080000:1.0:1787759403.094647:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759403.094651:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759403.094656:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759403.094666:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655317888/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759403.094675:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759403.094678:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787759403.094679:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759403.094682:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12459 00000100:00080000:1.0:1787759403.094684:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759403.094687:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759403.094691:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759403.094698:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655318016/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759403.094706:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759403.094708:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759403.094710:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759403.094712:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12459 00000100:00080000:1.0:1787759403.094715:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759403.094718:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759403.094720:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759403.094727:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655318144/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759403.095071:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655317248:192.168.201.107@tcp:250:kworker.0 00000800:00000100:0.0:1787759403.095140:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759403.095186:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759403/real 1787759403] req@ffff8c05b2d9a300 x1874588655317248/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759403.095204:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588655317248/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759403.095215:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759403.095219:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759403.095222:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655317248:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787759403.095229:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655317376:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759403.095251:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759403.095264:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759403/real 1787759403] req@ffff8c05b2d98380 x1874588655317376/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759403.095281:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655317376/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759403.095288:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759403.095291:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759403.095295:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655317376:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759403.095301:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655317504:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759403.095319:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759403.095330:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759403/real 1787759403] req@ffff8c05b2d9aa00 x1874588655317504/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759403.095342:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655317504/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759403.095349:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759403.095352:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759403.095355:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655317504:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759403.095362:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655317632:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759403.095381:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759403.095392:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759403/real 1787759403] req@ffff8c05b2d9b480 x1874588655317632/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759403.095403:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655317632/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759403.095417:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759403.095420:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759403.095424:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655317632:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759403.095430:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655317760:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759403.095448:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759403.095457:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759403/real 1787759403] req@ffff8c05b2d9b100 x1874588655317760/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759403.095468:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655317760/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759403.095476:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759403.095479:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759403.095482:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655317760:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759403.095489:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655317888:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759403.095507:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759403.095518:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759403/real 1787759403] req@ffff8c05b2d9bb80 x1874588655317888/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759403.095528:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655317888/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759403.095536:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759403.095539:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759403.095543:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655317888:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759403.095548:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655318016:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759403.095565:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759403.095574:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759403/real 1787759403] req@ffff8c05b2d99f80 x1874588655318016/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759403.095583:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655318016/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759403.095591:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759403.095594:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759403.095597:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655318016:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759403.095601:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655318144:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759403.095616:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759403.095624:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759403/real 1787759403] req@ffff8c05b2d98700 x1874588655318144/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759403.095634:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655318144/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759403.095641:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759403.095644:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759403.095647:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655318144:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759403.222054:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759403.670044:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759404.246030:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759404.694219:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759405.269972:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759405.718071:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759406.294025:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759406.742196:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759407.318018:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759407.766027:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787759408.213984:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759408.213993:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787759408.213997:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759408.214007:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12464 00000100:00080000:1.0:1787759408.214019:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759408.214027:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759408.214066:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655318272/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759408.214094:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759408.214097:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759408.214098:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759408.214101:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12464 00000100:00080000:1.0:1787759408.214103:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759408.214105:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759408.214113:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759408.214123:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655318400/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759408.214129:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759408.214132:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759408.214133:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759408.214137:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12464 00000100:00080000:1.0:1787759408.214140:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787759408.214148:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655318272:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787759408.214217:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:0.0:1787759408.214218:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:1.0:1787759408.214226:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759408.214241:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655318528/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759408.214256:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759408.214258:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759408.214260:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759408.214263:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12464 00000100:00080000:1.0:1787759408.214267:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759408.214271:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787759408.214272:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759408/real 1787759408] req@ffff8c05b2d98000 x1874588655318272/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787759408.214275:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759408.214293:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655318656/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759408.214299:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655318272/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759408.214301:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759408.214309:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759408.214312:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759408.214315:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12464 00000100:00080000:1.0:1787759408.214317:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759408.214319:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787759408.214321:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787759408.214322:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787759408.214327:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759408.214331:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655318272:192.168.201.107@tcp:250:kworker.0 00000100:00100000:1.0:1787759408.214335:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655318784/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759408.214339:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655318400:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787759408.214353:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759408.214357:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759408.214359:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759408.214364:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12464 00000100:00080000:1.0:1787759408.214369:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787759408.214370:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787759408.214377:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759408.214383:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759408.214388:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759408/real 1787759408] req@ffff8c05b2d99880 x1874588655318400/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759408.214396:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655318912/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759408.214403:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655318400/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759408.214412:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759408.214414:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759408.214417:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787759408.214419:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787759408.214421:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787759408.214423:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655318400:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787759408.214428:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12464 00000100:00080000:1.0:1787759408.214432:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759408.214435:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759408.214439:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759408.214448:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655319040/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759408.214463:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655318528:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787759408.214468:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759408.214473:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759408.214475:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759408.214480:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12464 00000100:00080000:1.0:1787759408.214484:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787759408.214488:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787759408.214492:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759408.214498:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759408.214510:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655319168/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759408.214518:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759408/real 1787759408] req@ffff8c05b2d98700 x1874588655318528/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759408.214544:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655318528/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759408.214549:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759408.214552:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759408.214554:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655318528:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759408.214559:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655318656:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759408.214573:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759408.214580:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759408/real 1787759408] req@ffff8c05b2d99f80 x1874588655318656/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759408.214586:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655318656/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759408.214591:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759408.214593:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759408.214595:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655318656:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759408.214598:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655318784:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759408.214609:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759408.214625:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759408/real 1787759408] req@ffff8c05b2d9bb80 x1874588655318784/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759408.214636:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655318784/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759408.214649:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759408.214652:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759408.214655:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655318784:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759408.214659:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655318912:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759408.214672:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759408.214679:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759408/real 1787759408] req@ffff8c05b2d9b100 x1874588655318912/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759408.214685:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655318912/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759408.214690:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759408.214692:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759408.214693:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655318912:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759408.214704:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655319040:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759408.214721:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759408.214727:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759408/real 1787759408] req@ffff8c05b2d9b480 x1874588655319040/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759408.214734:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655319040/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759408.214738:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759408.214740:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759408.214742:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655319040:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759408.214745:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655319168:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759408.214754:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759408.214759:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759408/real 1787759408] req@ffff8c05b2d9aa00 x1874588655319168/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759408.214765:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655319168/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759408.214770:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759408.214771:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759408.214773:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655319168:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759408.342128:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759408.790161:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759409.365991:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759409.814136:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759410.390025:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759410.838136:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759411.413983:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759411.862126:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759412.438016:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759412.886155:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787759413.334044:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759413.334056:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787759413.334061:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759413.334072:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12469 00000100:00080000:2.0:1787759413.334083:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759413.334090:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759413.334131:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588655319296/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759413.334159:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759413.334163:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759413.334165:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759413.334169:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12469 00000100:00080000:2.0:1787759413.334173:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759413.334177:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759413.334186:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759413.334205:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588655319424/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759413.334211:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655319296:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787759413.334221:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759413.334225:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759413.334228:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759413.334233:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12469 00000100:00080000:2.0:1787759413.334237:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759413.334241:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759413.334248:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759413.334260:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588655319552/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759413.334271:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759413.334276:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1787759413.334277:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787759413.334279:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759413.334285:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12469 00000100:00080000:2.0:1787759413.334288:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759413.334291:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759413.334296:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759413.334304:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588655319680/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759413.334311:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759413.334313:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759413.334314:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759413.334317:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12469 00000100:00080000:2.0:1787759413.334319:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759413.334322:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759413.334325:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759413.334332:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588655319808/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759413.334338:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787759413.334340:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759413/real 1787759413] req@ffff8c05b2e73100 x1874588655319296/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759413.334341:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759413.334344:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759413.334359:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12469 00000100:00100000:0.0:1787759413.334360:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588655319296/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759413.334364:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759413.334373:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787759413.334374:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787759413.334379:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787759413.334381:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759413.334387:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655319296:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787759413.334394:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588655319936/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759413.334404:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759413.334407:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787759413.334408:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759413.334412:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12469 00000100:00100000:0.0:1787759413.334416:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655319424:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787759413.334418:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759413.334424:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759413.334427:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759413.334438:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588655320064/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759413.334445:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787759413.334447:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787759413.334448:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759413.334452:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759413.334456:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12469 00000100:00080000:2.0:1787759413.334459:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759413.334461:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759413.334465:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759413.334468:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759413/real 1787759413] req@ffff8c05b2e72300 x1874588655319424/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759413.334475:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588655320192/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759413.334484:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588655319424/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759413.334492:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759413.334494:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759413.334496:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655319424:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759413.334501:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655319552:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759413.334516:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759413.334523:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759413/real 1787759413] req@ffff8c05b2e70a80 x1874588655319552/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759413.334531:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588655319552/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759413.334537:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759413.334540:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759413.334543:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655319552:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759413.334549:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655319680:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759413.334567:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759413.334578:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759413/real 1787759413] req@ffff8c05b2e70000 x1874588655319680/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759413.334585:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588655319680/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759413.334591:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759413.334593:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759413.334594:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655319680:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759413.334598:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655319808:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759413.334621:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759413.334631:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759413/real 1787759413] req@ffff8c05b2e70380 x1874588655319808/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759413.334641:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588655319808/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759413.334648:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759413.334651:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759413.334654:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655319808:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759413.334659:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655319936:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759413.334676:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759413.334684:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759413/real 1787759413] req@ffff8c05b2e72680 x1874588655319936/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759413.334690:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588655319936/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759413.334695:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759413.334697:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759413.334699:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655319936:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759413.334712:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655320064:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759413.334722:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759413.334727:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759413/real 1787759413] req@ffff8c05b2e71180 x1874588655320064/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759413.334733:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588655320064/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759413.334738:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759413.334740:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759413.334741:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655320064:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759413.334752:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655320192:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759413.334761:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759413.334767:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759413/real 1787759413] req@ffff8c05b2e72d80 x1874588655320192/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759413.334773:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588655320192/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759413.334778:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759413.334779:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759413.334781:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655320192:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759413.462009:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759413.910020:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759414.486021:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759414.934074:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759415.509968:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759415.958102:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759416.533997:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759416.982149:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759417.558040:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759418.006625:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787759418.454033:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759418.454046:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787759418.454051:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759418.454061:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12474 00000100:00080000:2.0:1787759418.454072:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759418.454079:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759418.454119:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588655320320/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759418.454149:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759418.454152:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759418.454154:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759418.454157:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12474 00000100:00080000:2.0:1787759418.454161:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759418.454164:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759418.454173:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759418.454189:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588655320448/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759418.454199:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759418.454202:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759418.454204:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759418.454210:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12474 00000100:00100000:0.0:1787759418.454211:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655320320:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787759418.454213:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759418.454220:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759418.454225:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759418.454236:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588655320576/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759418.454247:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759418.454250:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759418.454252:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759418.454256:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12474 00000100:00080000:2.0:1787759418.454260:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759418.454264:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759418.454276:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787759418.454287:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759418.454289:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588655320704/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759418.454299:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759418.454302:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759418.454304:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759418.454308:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12474 00000100:00080000:2.0:1787759418.454312:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759418.454324:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759418.454335:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759418.454347:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588655320832/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759418.454356:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787759418.454361:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759418/real 1787759418] req@ffff8c05b2e72a00 x1874588655320320/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759418.454366:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759418.454370:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759418.454376:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12474 00000100:00080000:2.0:1787759418.454380:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759418.454386:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759418.454390:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759418.454393:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588655320320/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759418.454407:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787759418.454407:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588655320960/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759418.454415:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759418.454423:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655320320:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787759418.454426:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759418.454432:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787759418.454434:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759418.454438:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12474 00000100:00080000:2.0:1787759418.454441:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787759418.454444:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655320448:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787759418.454448:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759418.454456:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759418.454470:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588655321088/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787759418.454479:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787759418.454483:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759418.454487:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759418.454490:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787759418.454496:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759418/real 1787759418] req@ffff8c05b2e70700 x1874588655320448/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759418.454496:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12474 00000100:00080000:2.0:1787759418.454501:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759418.454508:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759418.454515:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759418.454522:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588655320448/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759418.454529:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588655321216/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759418.454535:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759418.454544:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759418.454550:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655320448:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759418.454577:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655320576:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759418.454599:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759418.454609:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759418/real 1787759418] req@ffff8c05b2e73480 x1874588655320576/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759418.454628:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588655320576/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759418.454636:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759418.454639:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759418.454642:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655320576:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759418.454648:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655320704:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759418.454668:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759418.454678:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759418/real 1787759418] req@ffff8c05b2e71500 x1874588655320704/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759418.454689:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588655320704/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759418.454697:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759418.454708:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759418.454711:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655320704:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759418.454717:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655320832:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759418.454751:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759418.454763:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759418/real 1787759418] req@ffff8c05b2e73b80 x1874588655320832/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759418.454775:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588655320832/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759418.454793:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759418.454796:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759418.454800:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655320832:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759418.454805:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655320960:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759418.454824:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759418.454835:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759418/real 1787759418] req@ffff8c05b2e71c00 x1874588655320960/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759418.454846:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588655320960/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759418.454854:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759418.454857:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759418.454860:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655320960:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759418.454866:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655321088:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759418.454910:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759418.454923:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759418/real 1787759418] req@ffff8c05b2e71880 x1874588655321088/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759418.454933:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588655321088/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759418.454948:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759418.454952:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759418.454956:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655321088:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759418.454962:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655321216:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759418.454979:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759418.454989:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759418/real 1787759418] req@ffff8c05b2e70e00 x1874588655321216/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759418.455008:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588655321216/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759418.455015:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759418.455018:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759418.455021:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655321216:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759418.582004:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759419.030102:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759419.606136:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759420.055760:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759420.629975:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759421.078149:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759421.654018:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759422.102115:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759422.677957:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759423.126211:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787759423.573988:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759423.574000:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787759423.574005:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759423.574017:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12479 00000100:00080000:2.0:1787759423.574032:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759423.574042:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759423.574079:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73800 x1874588655321344/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759423.574121:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759423.574126:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759423.574127:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759423.574131:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12479 00000100:00080000:2.0:1787759423.574136:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759423.574140:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759423.574150:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759423.574166:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588655321472/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759423.574193:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759423.574197:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759423.574200:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759423.574204:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12479 00000100:00080000:2.0:1787759423.574209:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759423.574214:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787759423.574219:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655321344:192.168.201.107@tcp:250:kworker.0 00000008:00100000:2.0:1787759423.574221:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759423.574238:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588655321600/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759423.574250:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759423.574254:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759423.574256:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759423.574260:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12479 00000100:00080000:2.0:1787759423.574264:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759423.574268:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759423.574273:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787759423.574281:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759423.574287:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588655321728/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759423.574298:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759423.574301:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759423.574304:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759423.574308:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12479 00000100:00080000:2.0:1787759423.574312:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759423.574330:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787759423.574332:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759423/real 1787759423] req@ffff8c05b2e73800 x1874588655321344/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787759423.574337:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759423.574353:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588655321856/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759423.574358:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73800 x1874588655321344/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759423.574370:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759423.574375:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787759423.574377:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759423.574377:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759423.574384:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787759423.574386:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12479 00000100:00100000:0.0:1787759423.574391:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655321344:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787759423.574391:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759423.574399:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787759423.574403:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655321472:192.168.201.107@tcp:38:kworker.0 00000008:00100000:2.0:1787759423.574406:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759423.574424:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588655321984/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759423.574434:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787759423.574436:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787759423.574440:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787759423.574444:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759423.574449:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12479 00000100:00080000:2.0:1787759423.574453:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787759423.574456:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759423/real 1787759423] req@ffff8c05b2e70e00 x1874588655321472/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759423.574461:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759423.574469:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759423.574477:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588655321472/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759423.574485:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588655322112/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759423.574490:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759423.574499:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759423.574506:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655321472:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787759423.574514:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759423.574518:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759423.574520:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759423.574525:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12479 00000100:00080000:2.0:1787759423.574530:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759423.574534:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787759423.574538:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655321600:192.168.201.107@tcp:38:kworker.0 00000008:00100000:2.0:1787759423.574539:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759423.574557:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588655322240/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787759423.574565:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759423.574581:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759423/real 1787759423] req@ffff8c05b2e71880 x1874588655321600/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759423.574595:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588655321600/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759423.574603:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759423.574606:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759423.574610:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655321600:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759423.574616:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655321728:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759423.574631:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759423.574654:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759423/real 1787759423] req@ffff8c05b2e71c00 x1874588655321728/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759423.574671:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588655321728/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759423.574679:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759423.574682:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759423.574685:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655321728:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759423.574690:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655321856:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759423.574706:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759423.574715:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759423/real 1787759423] req@ffff8c05b2e73b80 x1874588655321856/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759423.574725:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588655321856/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759423.574733:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759423.574736:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759423.574738:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655321856:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759423.574743:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655321984:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759423.574757:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759423.574765:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759423/real 1787759423] req@ffff8c05b2e71500 x1874588655321984/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759423.574776:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588655321984/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759423.574789:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759423.574792:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759423.574795:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655321984:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759423.574800:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655322112:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759423.574813:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759423.574822:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759423/real 1787759423] req@ffff8c05b2e73480 x1874588655322112/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759423.574832:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588655322112/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759423.574840:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759423.574843:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759423.574846:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655322112:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759423.574862:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655322240:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759423.574903:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759423.574914:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759423/real 1787759423] req@ffff8c05b2e70700 x1874588655322240/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759423.574925:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588655322240/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759423.574934:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759423.574937:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759423.574941:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655322240:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759423.701994:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759424.150102:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759424.725995:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759425.174179:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759425.750734:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759426.198196:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759426.774034:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759427.222168:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759427.798040:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759428.246441:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787759428.694380:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759428.694393:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787759428.694396:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759428.694407:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12485 00000100:00080000:3.0:1787759428.694418:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759428.694426:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787759428.694469:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655322368/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759428.694499:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759428.694503:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759428.694505:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759428.694519:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12485 00000100:00080000:3.0:1787759428.694523:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759428.694527:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759428.694537:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759428.694559:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655322496/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759428.694569:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759428.694572:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759428.694574:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759428.694577:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12485 00000100:00080000:3.0:1787759428.694581:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759428.694584:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759428.694588:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759428.694598:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655322624/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759428.694608:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759428.694611:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759428.694613:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759428.694616:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12485 00000100:00080000:3.0:1787759428.694619:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759428.694623:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759428.694627:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759428.694651:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655322752/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759428.694660:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759428.694664:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759428.694665:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759428.694669:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12485 00000100:00080000:3.0:1787759428.694671:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759428.694675:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759428.694680:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759428.694691:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655322880/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759428.694705:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759428.694708:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759428.694709:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759428.694713:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12485 00000100:00080000:3.0:1787759428.694716:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759428.694719:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759428.694723:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759428.694733:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655323008/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759428.694741:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759428.694744:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787759428.694746:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759428.694749:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12485 00000100:00080000:3.0:1787759428.694752:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759428.694756:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759428.694760:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759428.694770:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655323136/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759428.694779:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759428.694782:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759428.694784:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759428.694787:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12485 00000100:00080000:3.0:1787759428.694790:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759428.694793:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759428.694797:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759428.694806:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655323264/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759428.695158:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655322368:192.168.201.107@tcp:250:kworker.0 00000800:00000100:0.0:1787759428.695230:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759428.695296:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759428/real 1787759428] req@ffff8c05a9f79180 x1874588655322368/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759428.695313:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655322368/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759428.695323:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759428.695327:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759428.695331:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655322368:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787759428.695339:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655322496:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759428.695360:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759428.695373:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759428/real 1787759428] req@ffff8c05a9f7bb80 x1874588655322496/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759428.695385:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655322496/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759428.695393:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759428.695396:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759428.695399:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655322496:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759428.695405:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655322624:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759428.695424:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759428.695435:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759428/real 1787759428] req@ffff8c05a9f79f80 x1874588655322624/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759428.695446:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655322624/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759428.695454:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759428.695457:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759428.695460:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655322624:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759428.695466:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655322752:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759428.695486:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759428.695496:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759428/real 1787759428] req@ffff8c05a9f7b100 x1874588655322752/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759428.695519:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655322752/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759428.695528:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759428.695531:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759428.695535:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655322752:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759428.695541:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655322880:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759428.695569:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759428.695580:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759428/real 1787759428] req@ffff8c05a9f7ad80 x1874588655322880/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759428.695592:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655322880/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759428.695600:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759428.695603:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759428.695606:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655322880:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759428.695612:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655323008:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759428.695629:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759428.695640:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759428/real 1787759428] req@ffff8c05a9f78e00 x1874588655323008/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759428.695650:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655323008/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759428.695658:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759428.695661:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759428.695664:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655323008:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759428.695669:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655323136:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759428.695688:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759428.695699:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759428/real 1787759428] req@ffff8c05a9f7b800 x1874588655323136/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759428.695710:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655323136/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759428.695718:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759428.695720:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759428.695724:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655323136:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759428.695730:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655323264:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759428.695748:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759428.695759:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759428/real 1787759428] req@ffff8c05a9f79500 x1874588655323264/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759428.695770:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655323264/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759428.695778:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759428.695781:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759428.695784:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655323264:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759428.822628:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759429.270437:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759429.846176:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759430.294305:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759430.870441:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759431.318262:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759431.894038:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759432.342305:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759432.918009:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759433.366162:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787759433.814087:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759433.814100:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787759433.814104:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759433.814116:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12490 00000100:00080000:3.0:1787759433.814128:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759433.814136:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787759433.814181:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655323392/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759433.814216:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759433.814220:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759433.814222:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759433.814227:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12490 00000100:00080000:3.0:1787759433.814231:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759433.814234:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759433.814245:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759433.814258:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655323520/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759433.814267:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759433.814270:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759433.814272:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759433.814276:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12490 00000100:00080000:3.0:1787759433.814279:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759433.814283:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759433.814288:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759433.814298:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655323648/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759433.814307:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759433.814310:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759433.814312:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759433.814315:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12490 00000100:00080000:3.0:1787759433.814318:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759433.814322:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759433.814327:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759433.814337:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655323776/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759433.814346:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759433.814349:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759433.814352:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759433.814355:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12490 00000100:00080000:3.0:1787759433.814358:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759433.814362:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759433.814366:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759433.814378:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655323904/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759433.814386:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759433.814389:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759433.814390:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759433.814394:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12490 00000100:00080000:3.0:1787759433.814397:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759433.814400:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759433.814404:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759433.814414:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655324032/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759433.814422:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759433.814425:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787759433.814426:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759433.814430:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12490 00000100:00080000:3.0:1787759433.814433:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759433.814441:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759433.814470:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759433.814482:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655324160/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759433.814490:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759433.814493:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759433.814494:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759433.814498:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12490 00000100:00080000:3.0:1787759433.814501:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759433.814505:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759433.814510:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759433.814522:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655324288/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759433.814533:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655323392:192.168.201.107@tcp:250:kworker.0 00000800:00000100:0.0:1787759433.814609:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759433.814663:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759433/real 1787759433] req@ffff8c05a9f7a300 x1874588655323392/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759433.814680:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655323392/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759433.814690:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759433.814695:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759433.814699:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655323392:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787759433.814706:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655323520:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759433.814729:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759433.814739:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759433/real 1787759433] req@ffff8c05a9f79880 x1874588655323520/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759433.814751:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655323520/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759433.814758:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759433.814762:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759433.814765:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655323520:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759433.814771:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655323648:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759433.814786:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759433.814794:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759433/real 1787759433] req@ffff8c05a9f78700 x1874588655323648/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759433.814805:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655323648/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759433.814812:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759433.814815:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759433.814818:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655323648:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759433.814824:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655323776:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759433.814838:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759433.814846:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759433/real 1787759433] req@ffff8c05a9f7b480 x1874588655323776/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759433.814863:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655323776/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759433.814871:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759433.814873:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759433.814904:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655323776:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759433.814910:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655323904:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759433.814927:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759433.814937:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759433/real 1787759433] req@ffff8c05a9f78380 x1874588655323904/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759433.814947:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655323904/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759433.814968:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759433.814971:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759433.814973:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655323904:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759433.814984:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655324032:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759433.815321:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759433.815335:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759433/real 1787759433] req@ffff8c05a9f78a80 x1874588655324032/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759433.815347:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655324032/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759433.815361:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759433.815363:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759433.815366:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655324032:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759433.815372:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655324160:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759433.815389:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759433.815398:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759433/real 1787759433] req@ffff8c05a9f7a680 x1874588655324160/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759433.815408:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655324160/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759433.815416:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759433.815418:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759433.815421:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655324160:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759433.815449:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655324288:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759433.815479:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759433.815491:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759433/real 1787759433] req@ffff8c05a9f7aa00 x1874588655324288/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759433.815501:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655324288/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759433.815509:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759433.815512:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759433.815515:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655324288:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759433.942040:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759434.390073:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759434.966577:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759435.414278:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759435.989999:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759436.438185:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759437.014147:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759437.462194:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759438.038019:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759438.486133:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787759438.934049:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759438.934061:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787759438.934066:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759438.934076:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12495 00000100:00080000:1.0:1787759438.934088:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759438.934095:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759438.934135:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655324416/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759438.934164:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759438.934167:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759438.934169:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759438.934172:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12495 00000100:00080000:1.0:1787759438.934176:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759438.934180:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759438.934191:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759438.934203:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588655324544/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759438.934212:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759438.934215:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759438.934217:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759438.934220:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12495 00000100:00080000:1.0:1787759438.934223:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759438.934227:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759438.934231:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759438.934244:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655324672/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759438.934252:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759438.934256:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759438.934258:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759438.934261:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12495 00000100:00080000:1.0:1787759438.934264:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759438.934268:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759438.934274:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759438.934282:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655324800/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759438.934292:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759438.934297:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1787759438.934298:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655324416:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787759438.934300:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759438.934309:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12495 00000100:00080000:1.0:1787759438.934313:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759438.934317:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759438.934323:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759438.934334:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655324928/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759438.934343:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759438.934347:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759438.934349:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787759438.934382:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759438.934433:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759438/real 1787759438] req@ffff8c05b2d98380 x1874588655324416/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759438.934452:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655324416/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759438.934462:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759438.934467:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759438.934471:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655324416:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787759438.934478:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655324544:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759438.934503:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759438.934516:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759438/real 1787759438] req@ffff8c05b2d9a300 x1874588655324544/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759438.934528:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588655324544/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759438.934536:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759438.934539:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759438.934542:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655324544:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759438.934548:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655324672:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759438.934569:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759438.934581:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759438/real 1787759438] req@ffff8c05b2d9aa00 x1874588655324672/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759438.934594:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655324672/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759438.934601:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759438.934604:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759438.934607:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655324672:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759438.934635:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655324800:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759438.934671:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759438.934683:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759438/real 1787759438] req@ffff8c05b2d9b480 x1874588655324800/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759438.934693:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655324800/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759438.934701:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759438.934704:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759438.934707:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655324800:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759438.934713:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655324928:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759438.934733:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759438.934745:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759438/real 1787759438] req@ffff8c05b2d9b100 x1874588655324928/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759438.934756:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655324928/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759438.934765:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759438.934767:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759438.934770:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655324928:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787759438.936527:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12495 00000100:00080000:1.0:1787759438.936537:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759438.936546:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759438.936555:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759438.936579:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655325056/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759438.936604:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759438.936608:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787759438.936610:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759438.936615:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12495 00000100:00080000:1.0:1787759438.936619:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759438.936623:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759438.936628:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759438.936642:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655325184/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759438.936651:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759438.936655:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759438.936656:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759438.936659:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12495 00000100:00080000:1.0:1787759438.936662:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759438.936664:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759438.936668:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759438.936678:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655325312/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759438.936685:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655325056:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759438.936729:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759438.936749:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759438/real 1787759438] req@ffff8c05b2d9bb80 x1874588655325056/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759438.936763:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655325056/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759438.936784:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759438.936788:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759438.936792:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655325056:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759438.936799:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655325184:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759438.936820:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759438.936831:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759438/real 1787759438] req@ffff8c05b2d99f80 x1874588655325184/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759438.936851:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655325184/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759438.936859:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759438.936863:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759438.936867:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655325184:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759438.936919:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655325312:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759438.936940:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759438.936950:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759438/real 1787759438] req@ffff8c05b2d98700 x1874588655325312/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759438.936962:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655325312/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759438.936980:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759438.936984:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759438.936988:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655325312:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759439.062012:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759439.510168:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759440.086037:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759440.534153:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759441.110116:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759441.558153:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759442.134012:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759442.582050:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759443.158029:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759443.606784:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787759444.054037:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759444.054048:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787759444.054052:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759444.054062:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12500 00000100:00080000:1.0:1787759444.054074:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759444.054081:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759444.054121:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655325440/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759444.054157:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759444.054160:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759444.054162:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759444.054165:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12500 00000100:00080000:1.0:1787759444.054169:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759444.054173:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759444.054182:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759444.054197:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655325568/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759444.054217:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759444.054220:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759444.054222:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759444.054226:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12500 00000100:00080000:1.0:1787759444.054229:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759444.054233:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759444.054238:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759444.054247:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655325696/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759444.054257:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759444.054260:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759444.054262:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759444.054273:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12500 00000100:00080000:1.0:1787759444.054289:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759444.054293:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759444.054298:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759444.054309:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655325824/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759444.054324:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759444.054328:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759444.054330:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759444.054333:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12500 00000100:00080000:1.0:1787759444.054336:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759444.054340:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759444.054344:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759444.054363:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655325952/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759444.054371:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759444.054375:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759444.054376:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759444.054380:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12500 00000100:00080000:1.0:1787759444.054383:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759444.054386:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759444.054391:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759444.054399:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655326080/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759444.054408:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759444.054412:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787759444.054413:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759444.054417:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12500 00000100:00080000:1.0:1787759444.054421:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759444.054425:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759444.054429:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759444.054438:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655326208/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759444.054446:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759444.054450:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759444.054451:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759444.054464:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12500 00000100:00080000:1.0:1787759444.054471:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759444.054475:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759444.054479:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759444.054490:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655326336/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759444.056355:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655325440:192.168.201.107@tcp:250:kworker.0 00000800:00000100:0.0:1787759444.056420:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759444.056469:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759444/real 1787759444] req@ffff8c05b2d99880 x1874588655325440/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759444.056487:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655325440/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759444.056497:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759444.056501:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759444.056505:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655325440:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787759444.056512:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655325568:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759444.056534:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759444.056546:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759444/real 1787759444] req@ffff8c05b2d98000 x1874588655325568/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759444.056556:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655325568/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759444.056565:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759444.056567:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759444.056570:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655325568:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759444.056576:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655325696:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759444.056594:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759444.056605:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759444/real 1787759444] req@ffff8c05b2d98700 x1874588655325696/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759444.056616:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655325696/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759444.056624:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759444.056627:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759444.056630:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655325696:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759444.056635:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655325824:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759444.056653:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759444.056663:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759444/real 1787759444] req@ffff8c05b2d99f80 x1874588655325824/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759444.056681:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655325824/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759444.056689:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759444.056692:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759444.056695:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655325824:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759444.056701:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655325952:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759444.056717:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759444.056727:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759444/real 1787759444] req@ffff8c05b2d9bb80 x1874588655325952/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759444.056739:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655325952/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759444.056747:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759444.056750:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759444.056753:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655325952:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759444.056759:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655326080:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759444.056776:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759444.056785:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759444/real 1787759444] req@ffff8c05b2d9b100 x1874588655326080/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759444.056795:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655326080/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759444.056809:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759444.056812:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759444.056816:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655326080:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759444.056822:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655326208:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759444.056838:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759444.056848:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759444/real 1787759444] req@ffff8c05b2d9b480 x1874588655326208/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759444.056858:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655326208/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759444.056866:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759444.056869:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759444.056872:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655326208:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759444.056911:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655326336:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759444.056930:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759444.056940:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759444/real 1787759444] req@ffff8c05b2d9aa00 x1874588655326336/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759444.056951:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655326336/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759444.056959:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759444.056962:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759444.056966:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655326336:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759444.182015:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759444.630079:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759445.206008:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759445.654290:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759446.229986:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759446.678213:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759447.254178:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759447.702485:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759448.277983:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759448.726111:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787759449.174034:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759449.174050:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787759449.174054:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759449.174065:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12505 00000100:00080000:1.0:1787759449.174076:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759449.174084:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759449.174120:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588655326464/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759449.174151:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759449.174156:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759449.174157:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759449.174161:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12505 00000100:00080000:1.0:1787759449.174164:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759449.174168:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759449.174176:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759449.174209:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655326592/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759449.174218:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759449.174221:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759449.174223:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759449.174226:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12505 00000100:00080000:1.0:1787759449.174230:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759449.174233:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759449.174238:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759449.174248:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655326720/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759449.174259:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787759449.174263:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655326464:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787759449.174264:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759449.174268:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759449.174274:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12505 00000100:00080000:1.0:1787759449.174277:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759449.174280:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759449.174286:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759449.174297:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655326848/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759449.174305:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759449.174308:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759449.174310:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759449.174314:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12505 00000100:00080000:1.0:1787759449.174318:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759449.174323:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:0.0:1787759449.174326:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:1.0:1787759449.174332:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759449.174346:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655326976/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759449.174355:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759449.174359:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759449.174361:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759449.174365:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12505 00000100:00080000:1.0:1787759449.174370:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787759449.174371:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759449/real 1787759449] req@ffff8c05b2d9a300 x1874588655326464/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759449.174378:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759449.174386:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759449.174390:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588655326464/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759449.174400:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787759449.174401:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655327104/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759449.174405:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759449.174412:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655326464:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787759449.174423:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655326592:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787759449.174423:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759449.174431:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787759449.174434:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759449.174439:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12505 00000100:00080000:1.0:1787759449.174443:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787759449.174445:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787759449.174450:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759449.174456:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759449.174457:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759449/real 1787759449] req@ffff8c05b2d98380 x1874588655326592/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759449.174476:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655326592/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759449.174476:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655327232/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759449.174499:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759449.174501:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759449.174504:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787759449.174508:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787759449.174509:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787759449.174517:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655326592:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787759449.174518:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12505 00000100:00080000:1.0:1787759449.174523:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759449.174528:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759449.174532:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759449.174543:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655327360/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759449.174550:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655326720:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759449.174577:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759449.174591:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759449/real 1787759449] req@ffff8c05b2d9aa00 x1874588655326720/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759449.174602:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655326720/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759449.174610:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759449.174613:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759449.174625:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655326720:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759449.174631:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655326848:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759449.174649:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759449.174660:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759449/real 1787759449] req@ffff8c05b2d9b480 x1874588655326848/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759449.174670:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655326848/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759449.174677:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759449.174680:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759449.174683:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655326848:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759449.174688:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655326976:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759449.174706:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759449.174717:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759449/real 1787759449] req@ffff8c05b2d9b100 x1874588655326976/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759449.174732:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655326976/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759449.174739:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759449.174742:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759449.174745:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655326976:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759449.174750:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655327104:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759449.174767:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759449.174774:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759449/real 1787759449] req@ffff8c05b2d9bb80 x1874588655327104/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759449.174781:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655327104/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759449.174786:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759449.174788:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759449.174790:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655327104:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759449.174801:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655327232:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759449.174813:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759449.174819:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759449/real 1787759449] req@ffff8c05b2d99f80 x1874588655327232/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759449.174825:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655327232/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759449.174833:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759449.174835:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759449.174837:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655327232:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759449.174855:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655327360:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759449.174865:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759449.174871:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759449/real 1787759449] req@ffff8c05b2d98700 x1874588655327360/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759449.174905:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655327360/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759449.174910:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759449.174912:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759449.174914:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655327360:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759449.302016:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759449.750138:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759450.325980:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759450.774181:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759451.350068:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759451.798158:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759452.373972:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759452.822204:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759453.397985:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759453.846256:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787759454.293997:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759454.294008:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787759454.294014:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759454.294026:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12510 00000100:00080000:1.0:1787759454.294037:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759454.294045:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759454.294086:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655327488/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759454.294137:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759454.294141:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759454.294151:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759454.294156:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12510 00000100:00080000:1.0:1787759454.294161:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759454.294166:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759454.294177:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759454.294193:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655327616/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759454.294197:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655327488:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787759454.294212:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759454.294216:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759454.294217:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759454.294222:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12510 00000100:00080000:1.0:1787759454.294227:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759454.294231:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759454.294237:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759454.294252:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655327744/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759454.294263:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759454.294267:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759454.294269:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759454.294273:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12510 00000100:00080000:1.0:1787759454.294277:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787759454.294281:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787759454.294285:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759454.294292:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759454.294305:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655327872/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759454.294315:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759454.294318:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759454.294320:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759454.294324:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12510 00000100:00080000:1.0:1787759454.294329:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759454.294334:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787759454.294338:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759454/real 1787759454] req@ffff8c05b2d98000 x1874588655327488/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787759454.294341:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759454.294360:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655328000/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759454.294365:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655327488/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759454.294378:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759454.294382:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787759454.294383:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759454.294385:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759454.294391:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787759454.294393:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12510 00000100:00100000:0.0:1787759454.294400:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655327488:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787759454.294400:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759454.294408:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759454.294414:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759454.294426:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655328128/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759454.294433:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655327616:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787759454.294450:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759454.294454:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787759454.294456:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759454.294460:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12510 00000100:00080000:1.0:1787759454.294464:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759454.294468:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759454.294473:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787759454.294475:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759454.294490:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655328256/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759454.294494:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759454/real 1787759454] req@ffff8c05b2d99880 x1874588655327616/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759454.294517:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787759454.294519:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655327616/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759454.294526:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759454.294531:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759454.294532:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759454.294537:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12510 00000100:00080000:0.0:1787759454.294539:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787759454.294545:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787759454.294546:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655327616:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787759454.294553:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787759454.294557:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655327744:192.168.201.107@tcp:38:kworker.0 00000008:00100000:1.0:1787759454.294561:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759454.294578:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655328384/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787759454.294584:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759454.294615:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759454/real 1787759454] req@ffff8c05b2d98700 x1874588655327744/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759454.294629:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655327744/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759454.294638:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759454.294641:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759454.294645:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655327744:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759454.294650:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655327872:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759454.294668:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759454.294678:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759454/real 1787759454] req@ffff8c05b2d99f80 x1874588655327872/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759454.294689:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655327872/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759454.294695:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759454.294698:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759454.294701:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655327872:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759454.294707:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655328000:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759454.294724:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759454.294736:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759454/real 1787759454] req@ffff8c05b2d9bb80 x1874588655328000/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759454.294747:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655328000/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759454.294754:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759454.294756:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759454.294759:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655328000:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759454.294776:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655328128:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759454.294794:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759454.294806:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759454/real 1787759454] req@ffff8c05b2d9b100 x1874588655328128/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759454.294817:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655328128/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759454.294825:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759454.294828:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759454.294831:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655328128:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759454.294837:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655328256:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759454.294855:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759454.294866:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759454/real 1787759454] req@ffff8c05b2d9b480 x1874588655328256/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759454.294914:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655328256/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759454.294924:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759454.294927:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759454.294930:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655328256:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759454.294937:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655328384:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759454.294957:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759454.294969:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759454/real 1787759454] req@ffff8c05b2d9aa00 x1874588655328384/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759454.294980:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655328384/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759454.294988:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759454.294991:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759454.294995:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655328384:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759454.421994:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759454.870131:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759455.445986:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759455.894224:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759456.469985:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759456.918414:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759457.493936:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759457.942325:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759458.517993:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759458.966129:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787759459.414017:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759459.414030:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787759459.414035:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759459.414044:0:125331:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12515 00000100:00080000:1.0:1787759459.414053:0:125331:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759459.414060:0:125331:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759459.414092:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655328512/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759459.414122:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759459.414125:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759459.414127:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759459.414131:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12515 00000100:00080000:1.0:1787759459.414143:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759459.414149:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759459.414149:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655328512:192.168.201.107@tcp:250:kworker.0 00000008:00100000:1.0:1787759459.414163:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759459.414178:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588655328640/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759459.414186:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759459.414189:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759459.414191:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759459.414195:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12515 00000100:00080000:1.0:1787759459.414200:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759459.414204:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759459.414210:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787759459.414219:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759459.414225:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655328768/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759459.414244:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759459.414247:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759459.414249:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759459.414253:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12515 00000100:00080000:1.0:1787759459.414257:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759459.414260:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759459.414264:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759459.414272:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759459/real 1787759459] req@ffff8c05b2d98380 x1874588655328512/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759459.414274:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655328896/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759459.414295:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759459.414297:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759459.414297:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759459.414300:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12515 00000100:00080000:1.0:1787759459.414302:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759459.414304:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759459.414307:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759459.414308:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655328512/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759459.414319:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655329024/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759459.414319:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759459.414324:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759459.414330:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655328512:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787759459.414338:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759459.414342:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759459.414344:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759459.414349:0:125331:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12515 00000100:00080000:1.0:1787759459.414353:0:125331:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759459.414357:0:125331:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759459.414360:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655328640:192.168.201.107@tcp:38:kworker.0 00000008:00100000:1.0:1787759459.414363:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759459.414376:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655329152/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787759459.414384:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787759459.414391:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759459.414396:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787759459.414398:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759459/real 1787759459] req@ffff8c05b2d9a300 x1874588655328640/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759459.414399:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759459.414409:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12515 00000100:00100000:2.0:1787759459.414413:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588655328640/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759459.414415:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759459.414424:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759459.414425:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:2.0:1787759459.414435:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759459.414438:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655328640:192.168.201.107@tcp:38:kworker.0 00000008:00100000:1.0:1787759459.414441:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759459.414443:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655328768:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759459.414461:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:1.0:1787759459.414465:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655329280/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759459.414475:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759459/real 1787759459] req@ffff8c05b2d9aa00 x1874588655328768/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759459.414476:0:125331:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759459.414484:0:125331:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759459.414489:0:125331:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759459.414495:0:125331:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12515 00000100:00080000:1.0:1787759459.414499:0:125331:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787759459.414501:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655328768/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759459.414504:0:125331:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759459.414511:0:125331:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787759459.414516:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759459.414521:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:1.0:1787759459.414524:0:125331:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655329408/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759459.414527:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655328768:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759459.414539:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655328896:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759459.414571:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759459.414589:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759459/real 1787759459] req@ffff8c05b2d9b480 x1874588655328896/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759459.414601:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655328896/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759459.414615:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759459.414619:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759459.414622:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655328896:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759459.414627:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655329024:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759459.414655:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759459.414666:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759459/real 1787759459] req@ffff8c05b2d9b100 x1874588655329024/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759459.414676:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655329024/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759459.414684:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759459.414686:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759459.414689:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655329024:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759459.414706:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655329152:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759459.414722:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759459.414730:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759459/real 1787759459] req@ffff8c05b2d9bb80 x1874588655329152/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759459.414736:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655329152/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759459.414744:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759459.414746:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759459.414750:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655329152:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759459.414755:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655329280:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759459.414774:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759459.414783:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759459/real 1787759459] req@ffff8c05b2d99f80 x1874588655329280/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759459.414794:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655329280/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759459.414799:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759459.414801:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759459.414803:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655329280:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759459.414807:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655329408:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759459.414819:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759459.414825:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759459/real 1787759459] req@ffff8c05b2d98700 x1874588655329408/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759459.414832:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655329408/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759459.414836:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759459.414838:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759459.414839:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655329408:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759459.541997:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759459.990102:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759460.566008:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759461.014135:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759461.589989:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759462.038224:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759462.614030:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759463.062094:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759463.637977:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759464.086137:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787759464.533980:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759464.533990:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787759464.533995:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759464.534006:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12520 00000100:00080000:3.0:1787759464.534019:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759464.534028:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787759464.534073:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655329536/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759464.534105:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759464.534109:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759464.534111:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759464.534116:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12520 00000100:00080000:3.0:1787759464.534119:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759464.534123:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759464.534132:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759464.534145:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655329664/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759464.534155:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655329536:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787759464.534160:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759464.534172:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759464.534174:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759464.534182:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12520 00000100:00080000:3.0:1787759464.534186:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759464.534191:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759464.534196:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759464.534208:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655329792/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759464.534219:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759464.534223:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759464.534225:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759464.534229:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12520 00000100:00080000:3.0:1787759464.534232:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759464.534236:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759464.534241:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759464.534253:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655329920/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759464.534264:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759464.534268:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759464.534270:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759464.534274:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12520 00000100:00080000:3.0:1787759464.534278:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759464.534283:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759464.534289:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759464.534291:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655329664:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759464.534303:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655330048/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759464.534316:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759464.534319:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759464.534321:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759464.534325:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12520 00000100:00080000:3.0:1787759464.534329:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787759464.534333:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655329792:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787759464.534335:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759464.534344:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759464.534355:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655330176/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759464.534365:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759464.534368:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787759464.534371:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759464.534375:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12520 00000100:00080000:3.0:1787759464.534379:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787759464.534380:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655329920:192.168.201.107@tcp:8:kworker.0 00000100:00080000:3.0:1787759464.534387:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759464.534394:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759464.534405:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655330048:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759464.534409:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655330304/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759464.534421:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759464.534425:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759464.534427:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759464.534432:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12520 00000100:00100000:2.0:1787759464.534437:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655330176:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787759464.534437:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759464.534443:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759464.534450:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759464.534462:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655330432/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759464.534476:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655330304:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759464.534505:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655330432:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759464.662006:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759465.110088:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759465.685955:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759466.134163:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759466.709991:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759467.158041:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787759467.606125:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.107:988 00000400:00000100:0.0:1787759467.606168:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.107@tcp at host 192.168.201.107 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787759467.606186:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787759467.606237:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787759467.606256:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787759467.606263:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787759467.606272:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787759467.606280:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787759467.606288:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:0.0:1787759467.606296:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:2.0:1787759467.606473:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759464/real 1787759467] req@ffff8c05a9f7a300 x1874588655330432/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759467.606509:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655330432/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759467.606522:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759467.606528:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759467.606533:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655330432:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759467.606541:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759464/real 1787759467] req@ffff8c05a9f79880 x1874588655330304/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759467.606552:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655330304/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759467.606560:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759467.606563:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759467.606568:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655330304:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759467.606575:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759464/real 1787759467] req@ffff8c05a9f7b480 x1874588655330048/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759467.606587:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655330048/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759467.606602:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759467.606606:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759467.606609:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655330048:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759467.606614:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759464/real 1787759467] req@ffff8c05a9f78700 x1874588655330176/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759467.606625:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655330176/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759467.606632:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759467.606635:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759467.606638:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655330176:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759467.606644:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759464/real 1787759467] req@ffff8c05a9f7a680 x1874588655329664/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759467.606654:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655329664/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759467.606661:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759467.606664:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759467.606668:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655329664:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759467.606674:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759464/real 1787759467] req@ffff8c05a9f78a80 x1874588655329792/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759467.606690:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655329792/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759467.606697:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759467.606700:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759467.606703:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655329792:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759467.606710:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759464/real 1787759467] req@ffff8c05a9f78380 x1874588655329920/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759467.606721:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655329920/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759467.606728:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759467.606732:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759467.606735:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655329920:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759467.606741:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759464/real 1787759467] req@ffff8c05a9f7aa00 x1874588655329536/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759467.606752:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655329536/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759467.606759:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759467.606763:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759467.606766:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655329536:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787759467.734930:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759468.182089:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759468.758015:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759469.206038:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787759469.654020:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759469.654030:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787759469.654035:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759469.654047:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12526 00000100:00080000:2.0:1787759469.654059:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759469.654067:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759469.654113:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588655330560/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759469.654142:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759469.654145:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759469.654147:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759469.654151:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12526 00000100:00080000:2.0:1787759469.654155:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759469.654159:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759469.654168:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759469.654192:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655330560:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787759469.654195:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588655330688/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759469.654209:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759469.654212:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759469.654214:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787759469.654261:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787759469.654276:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12526 00000100:00080000:2.0:1787759469.654282:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759469.654286:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759469.654292:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759469.654305:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588655330816/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759469.654311:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759469/real 1787759469] req@ffff8c05b2e72a00 x1874588655330560/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759469.654320:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759469.654328:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759469.654330:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759469.654339:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12526 00000100:00100000:0.0:1787759469.654343:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588655330560/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759469.654345:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759469.654354:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787759469.654362:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787759469.654363:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787759469.654369:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759469.654376:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655330560:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787759469.654379:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588655330944/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759469.654391:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759469.654395:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759469.654396:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759469.654400:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12526 00000100:00100000:0.0:1787759469.654405:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655330688:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787759469.654405:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759469.654412:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759469.654418:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759469.654430:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588655331072/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787759469.654436:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787759469.654447:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759469.654451:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787759469.654453:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759469/real 1787759469] req@ffff8c05b2e72d80 x1874588655330688/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759469.654454:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759469.654462:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12526 00000100:00080000:2.0:1787759469.654468:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759469.654473:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787759469.654478:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588655330688/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787759469.654479:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787759469.654492:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787759469.654493:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588655331200/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759469.654500:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759469.654508:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655330688:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787759469.654513:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787759469.654518:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655330816:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787759469.654519:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787759469.654523:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759469.654529:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12526 00000100:00080000:2.0:1787759469.654533:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759469.654537:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:0.0:1787759469.654543:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:2.0:1787759469.654553:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759469.654555:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759469/real 1787759469] req@ffff8c05b2e71180 x1874588655330816/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759469.654567:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588655331328/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759469.654594:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787759469.654596:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588655330816/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759469.654600:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759469.654603:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759469.654611:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12526 00000100:00080000:0.0:1787759469.654615:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759469.654617:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759469.654623:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:2.0:1787759469.654625:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787759469.654631:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655330816:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759469.654638:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655330944:192.168.201.107@tcp:8:kworker.0 00000008:00100000:2.0:1787759469.654642:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759469.654658:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588655331456/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787759469.654662:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759469.654675:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759469/real 1787759469] req@ffff8c05b2e72680 x1874588655330944/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759469.654695:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588655330944/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759469.654704:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759469.654707:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759469.654710:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655330944:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759469.654722:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655331072:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759469.654738:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759469.654747:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759469/real 1787759469] req@ffff8c05b2e70380 x1874588655331072/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759469.654758:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588655331072/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759469.654765:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759469.654768:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759469.654772:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655331072:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759469.654777:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655331200:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759469.654790:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759469.654798:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759469/real 1787759469] req@ffff8c05b2e70000 x1874588655331200/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759469.654808:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588655331200/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759469.654822:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759469.654837:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759469.654840:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655331200:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759469.654846:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655331328:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759469.654862:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759469.654871:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759469/real 1787759469] req@ffff8c05b2e70a80 x1874588655331328/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759469.654901:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588655331328/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759469.654910:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759469.654913:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759469.654916:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655331328:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759469.654922:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655331456:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759469.654938:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759469.654947:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759469/real 1787759469] req@ffff8c05b2e72300 x1874588655331456/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759469.654958:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588655331456/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759469.654966:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759469.654977:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759469.654981:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655331456:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759469.781973:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759470.230153:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759470.805981:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759471.254593:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759471.829985:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759472.278088:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759472.854016:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759473.302107:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759473.878028:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759474.326203:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787759474.774008:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759474.774018:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787759474.774027:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759474.774038:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12531 00000100:00080000:3.0:1787759474.774048:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759474.774054:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787759474.774098:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655331584/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759474.774126:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759474.774129:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759474.774130:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759474.774134:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12531 00000100:00080000:3.0:1787759474.774137:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759474.774140:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759474.774153:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759474.774167:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655331712/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759474.774175:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759474.774179:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759474.774181:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759474.774185:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12531 00000100:00080000:3.0:1787759474.774189:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759474.774194:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759474.774199:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759474.774212:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655331840/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759474.774224:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655331584:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787759474.774228:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759474.774240:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759474.774241:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759474.774246:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12531 00000100:00080000:3.0:1787759474.774248:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759474.774252:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759474.774257:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759474.774266:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655331968/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759474.774273:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759474.774275:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759474.774276:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759474.774282:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12531 00000100:00080000:3.0:1787759474.774285:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759474.774287:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759474.774289:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759474.774298:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655332096/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759474.774305:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787759474.774312:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787759474.774313:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759474.774317:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759474.774321:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12531 00000100:00080000:3.0:1787759474.774324:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759474.774327:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759474.774330:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759474.774339:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655332224/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759474.774345:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759474.774348:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787759474.774349:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759474.774352:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12531 00000100:00080000:3.0:1787759474.774355:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759474.774357:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759474.774361:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759474.774368:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655332352/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759474.774382:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759474.774384:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759474.774386:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759474.774388:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12531 00000100:00080000:3.0:1787759474.774392:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787759474.774398:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759474/real 1787759474] req@ffff8c05a9f79500 x1874588655331584/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759474.774398:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759474.774408:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759474.774427:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655332480/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759474.774441:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655331584/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759474.774466:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759474.774470:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759474.774474:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655331584:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787759474.774484:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655331712:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759474.774520:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759474.774533:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759474/real 1787759474] req@ffff8c05a9f7b800 x1874588655331712/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759474.774545:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655331712/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759474.774553:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759474.774557:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759474.774560:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655331712:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759474.774585:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655331840:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759474.774608:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759474.774618:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759474/real 1787759474] req@ffff8c05a9f78e00 x1874588655331840/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759474.774629:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655331840/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759474.774636:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759474.774639:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759474.774642:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655331840:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759474.774649:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655331968:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759474.774667:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759474.774676:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759474/real 1787759474] req@ffff8c05a9f7ad80 x1874588655331968/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759474.774686:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655331968/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759474.774693:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759474.774697:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759474.774700:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655331968:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759474.774706:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655332096:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759474.774723:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759474.774749:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759474/real 1787759474] req@ffff8c05a9f7b100 x1874588655332096/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759474.774768:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655332096/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759474.774776:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759474.774779:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759474.774783:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655332096:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759474.774789:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655332224:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759474.774809:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759474.774820:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759474/real 1787759474] req@ffff8c05a9f79f80 x1874588655332224/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759474.774842:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655332224/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759474.774849:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759474.774853:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759474.774856:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655332224:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759474.774862:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655332352:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759474.774923:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759474.774935:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759474/real 1787759474] req@ffff8c05a9f7bb80 x1874588655332352/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759474.774946:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655332352/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759474.774960:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759474.774963:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759474.774967:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655332352:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759474.774973:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655332480:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759474.774992:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759474.775004:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759474/real 1787759474] req@ffff8c05a9f79180 x1874588655332480/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759474.775014:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655332480/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759474.775022:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759474.775026:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759474.775029:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655332480:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759474.902000:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759475.350055:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759475.926004:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759476.374106:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759476.949982:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759477.398053:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759477.974780:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759478.422120:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759478.998052:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759479.446207:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787759479.894016:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759479.894027:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787759479.894039:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759479.894050:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12536 00000100:00080000:3.0:1787759479.894069:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759479.894078:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787759479.894143:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655332608/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759479.894178:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759479.894182:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759479.894190:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759479.894196:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12536 00000100:00080000:3.0:1787759479.894201:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759479.894206:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759479.894223:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759479.894233:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655332608:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787759479.894245:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655332736/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759479.894257:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759479.894267:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759479.894269:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759479.894273:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12536 00000100:00080000:3.0:1787759479.894277:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759479.894281:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759479.894297:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759479.894310:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655332864/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787759479.894325:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787759479.894334:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759479.894345:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759479.894347:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759479.894353:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12536 00000100:00080000:3.0:1787759479.894357:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759479.894362:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759479.894373:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759479.894386:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655332992/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759479.894393:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759479/real 1787759479] req@ffff8c05a9f79180 x1874588655332608/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759479.894423:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759479.894429:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1787759479.894430:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655332608/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759479.894432:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759479.894446:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12536 00000100:00080000:0.0:1787759479.894449:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759479.894451:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759479.894457:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787759479.894460:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787759479.894468:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655332608:192.168.201.107@tcp:250:kworker.0 00000008:00100000:3.0:1787759479.894469:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759479.894486:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655333120/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759479.894496:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759479.894500:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787759479.894504:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655332736:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787759479.894512:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759479.894517:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12536 00000100:00080000:3.0:1787759479.894522:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759479.894527:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:0.0:1787759479.894537:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:3.0:1787759479.894541:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759479.894558:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759479/real 1787759479] req@ffff8c05a9f7bb80 x1874588655332736/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759479.894563:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655333248/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759479.894577:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655332736/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759479.894592:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759479.894594:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759479.894599:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759479.894604:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655332736:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787759479.894608:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787759479.894614:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655332864:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787759479.894618:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759479.894623:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12536 00000100:00080000:3.0:1787759479.894629:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787759479.894664:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787759479.894664:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759479.894673:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759479.894683:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759479/real 1787759479] req@ffff8c05a9f79f80 x1874588655332864/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759479.894695:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655333376/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759479.894699:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655332864/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759479.894717:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759479.894721:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787759479.894723:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787759479.894724:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655332864:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787759479.894732:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787759479.894734:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655332992:192.168.201.107@tcp:8:kworker.0 00000100:00080000:3.0:1787759479.894735:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759479.894748:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12536 00000100:00080000:3.0:1787759479.894752:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759479.894756:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759479.894762:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787759479.894763:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759479.894774:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759479/real 1787759479] req@ffff8c05a9f7b100 x1874588655332992/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759479.894789:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655332992/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759479.894794:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655333504/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759479.894798:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759479.894806:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759479.894819:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655332992:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759479.894827:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655333120:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759479.894844:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759479.894854:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759479/real 1787759479] req@ffff8c05a9f7ad80 x1874588655333120/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759479.894872:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655333120/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759479.894909:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759479.894913:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759479.894917:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655333120:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759479.894934:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655333248:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759479.894965:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759479.894976:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759479/real 1787759479] req@ffff8c05a9f78e00 x1874588655333248/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759479.894987:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655333248/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759479.894995:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759479.894998:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759479.895015:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655333248:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759479.895022:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655333376:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759479.895040:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759479.895052:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759479/real 1787759479] req@ffff8c05a9f7b800 x1874588655333376/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759479.895062:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655333376/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759479.895069:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759479.895073:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759479.895076:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655333376:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759479.895091:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655333504:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759479.895110:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759479.895121:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759479/real 1787759479] req@ffff8c05a9f79500 x1874588655333504/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759479.895131:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655333504/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759479.895138:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759479.895141:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759479.895146:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655333504:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759480.022015:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759480.470205:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759481.046032:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759481.494042:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759482.070012:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759482.518272:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759483.094537:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759483.542239:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759484.118288:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759484.566202:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787759485.013992:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759485.014005:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787759485.014009:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759485.014020:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12541 00000100:00080000:0.0:1787759485.014030:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759485.014038:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787759485.014080:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74a80 x1874588655333632/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759485.014107:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759485.014111:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787759485.014113:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759485.014117:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12541 00000100:00080000:0.0:1787759485.014121:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759485.014124:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759485.014132:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759485.014565:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655333760/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759485.014577:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759485.014581:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787759485.014618:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655333632:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787759485.014636:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759485.014642:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12541 00000100:00080000:0.0:1787759485.014648:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759485.014653:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759485.014659:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759485.014673:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655333888/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787759485.014677:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787759485.014688:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759485.014692:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787759485.014693:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759485.014698:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12541 00000100:00080000:0.0:1787759485.014702:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759485.014705:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759485.014710:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759485.014721:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759485/real 1787759485] req@ffff8c05a9f74a80 x1874588655333632/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759485.014724:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588655334016/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759485.014743:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787759485.014745:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74a80 x1874588655333632/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759485.014747:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787759485.014751:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759485.014759:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12541 00000100:00080000:2.0:1787759485.014760:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759485.014766:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759485.014766:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759485.014772:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655333632:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787759485.014775:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759485.014780:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655333760:192.168.201.107@tcp:38:kworker.0 00000008:00100000:0.0:1787759485.014783:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759485.014798:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655334144/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787759485.014802:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759485.014812:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759485/real 1787759485] req@ffff8c05a9f74380 x1874588655333760/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759485.014833:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655333760/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759485.014834:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759485.014843:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759485.014847:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759485.014848:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759485.014852:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787759485.014858:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12541 00000100:00100000:2.0:1787759485.014858:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655333760:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787759485.014867:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759485.014871:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759485.014910:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759485.014913:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655333888:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759485.014928:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588655334272/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787759485.014932:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787759485.014945:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787759485.014947:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759485/real 1787759485] req@ffff8c05a9f76680 x1874588655333888/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759485.014951:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787759485.014955:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759485.014965:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12541 00000100:00080000:0.0:1787759485.014970:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787759485.014972:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655333888/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759485.014976:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759485.014982:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787759485.014989:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759485.014995:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759485.014996:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655334400/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759485.015004:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655333888:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787759485.015007:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759485.015011:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787759485.015013:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787759485.015013:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655334016:192.168.201.107@tcp:8:kworker.0 00000100:00080000:0.0:1787759485.015021:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12541 00000100:00080000:0.0:1787759485.015025:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759485.015027:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759485.015031:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787759485.015039:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759485.015040:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588655334528/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759485.015050:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759485/real 1787759485] req@ffff8c05a9f77b80 x1874588655334016/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759485.015059:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588655334016/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759485.015069:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759485.015072:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759485.015074:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655334016:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759485.015077:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655334144:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759485.015088:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759485.015096:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759485/real 1787759485] req@ffff8c05a9f77800 x1874588655334144/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759485.015103:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655334144/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759485.015108:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759485.015111:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759485.015113:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655334144:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759485.015126:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655334272:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759485.015151:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759485.015157:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759485/real 1787759485] req@ffff8c05a9f76a00 x1874588655334272/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759485.015164:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588655334272/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759485.015168:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759485.015170:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759485.015172:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655334272:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759485.015176:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655334400:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759485.015185:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759485.015190:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759485/real 1787759485] req@ffff8c05a9f76300 x1874588655334400/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759485.015196:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655334400/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759485.015201:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759485.015202:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759485.015204:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655334400:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759485.015206:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655334528:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759485.015214:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759485.015219:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759485/real 1787759485] req@ffff8c05a9f75c00 x1874588655334528/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759485.015225:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588655334528/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759485.015229:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759485.015230:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759485.015232:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655334528:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759485.142029:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759485.591297:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759486.166618:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759486.614131:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759487.190017:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759487.638237:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759488.214019:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759488.663371:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759489.238006:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759489.686570:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787759490.134160:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759490.134172:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787759490.134177:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759490.134188:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12546 00000100:00080000:2.0:1787759490.134200:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759490.134208:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759490.134251:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73100 x1874588655334656/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759490.134291:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759490.134295:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759490.134298:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759490.134302:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12546 00000100:00080000:2.0:1787759490.134305:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759490.134309:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759490.134319:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759490.134334:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588655334784/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787759490.134341:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655334656:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787759490.134347:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759490.134351:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759490.134353:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759490.134357:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12546 00000100:00080000:2.0:1787759490.134360:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759490.134364:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759490.134368:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759490.134378:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588655334912/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759490.134388:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759490.134392:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759490.134393:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759490.134397:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12546 00000100:00080000:2.0:1787759490.134403:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759490.134407:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759490.134412:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759490.134421:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588655335040/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759490.134429:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759490.134432:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759490.134434:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759490.134437:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12546 00000100:00080000:2.0:1787759490.134441:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759490.134445:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759490.134449:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759490.134458:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588655335168/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759490.135191:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759490.135195:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759490.135197:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759490.135202:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12546 00000100:00080000:2.0:1787759490.135208:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759490.135212:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759490.135217:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759490.135233:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588655335296/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759490.135242:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759490.135245:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787759490.135247:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759490.135250:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12546 00000100:00080000:2.0:1787759490.135253:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759490.135257:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759490.135262:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759490.135272:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588655335424/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759490.135280:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759490.135283:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759490.135284:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759490.135287:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12546 00000100:00080000:2.0:1787759490.135291:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759490.135294:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759490.135298:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759490.135307:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588655335552/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787759490.135697:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759490.135852:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759490/real 1787759490] req@ffff8c05b2e73100 x1874588655334656/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759490.135872:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73100 x1874588655334656/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759490.135893:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759490.135897:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759490.135900:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655334656:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787759490.135925:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655334784:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787759490.135965:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759490.135974:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759490/real 1787759490] req@ffff8c05b2e72300 x1874588655334784/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759490.135982:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588655334784/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759490.135988:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759490.135990:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759490.135992:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655334784:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759490.135997:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655334912:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787759490.136007:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759490.136013:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759490/real 1787759490] req@ffff8c05b2e70a80 x1874588655334912/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759490.136021:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588655334912/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759490.136030:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759490.136032:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759490.136034:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655334912:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759490.136038:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655335040:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787759490.136048:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759490.136053:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759490/real 1787759490] req@ffff8c05b2e70000 x1874588655335040/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759490.136059:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588655335040/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759490.136065:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759490.136066:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759490.136068:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655335040:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759490.136074:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655335168:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787759490.136091:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759490.136100:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759490/real 1787759490] req@ffff8c05b2e70380 x1874588655335168/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759490.136112:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588655335168/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759490.136121:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759490.136124:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759490.136128:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655335168:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759490.136134:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655335296:192.168.201.107@tcp:38:kworker.0 00000800:00000100:3.0:1787759490.136154:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759490.136165:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759490/real 1787759490] req@ffff8c05b2e72680 x1874588655335296/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759490.136176:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588655335296/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759490.136182:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759490.136185:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759490.136187:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655335296:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759490.136193:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655335424:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787759490.136210:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759490.136221:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759490/real 1787759490] req@ffff8c05b2e71180 x1874588655335424/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759490.136230:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588655335424/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759490.136238:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759490.136241:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759490.136244:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655335424:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759490.136249:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655335552:192.168.201.107@tcp:8:kworker.0 00000800:00000100:3.0:1787759490.136267:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759490.136277:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759490/real 1787759490] req@ffff8c05b2e72d80 x1874588655335552/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759490.136288:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588655335552/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759490.136295:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759490.136298:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759490.136301:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655335552:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759490.262142:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759490.710269:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759491.287485:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759491.734114:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759492.310031:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759492.758114:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759493.333985:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759493.783469:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759494.357984:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759494.809623:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787759495.254912:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759495.254923:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787759495.254929:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759495.254943:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12551 00000100:00080000:3.0:1787759495.254957:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759495.254966:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787759495.255011:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655335680/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759495.255043:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759495.255046:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759495.255048:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759495.255054:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12551 00000100:00080000:3.0:1787759495.255058:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759495.255063:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759495.255074:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759495.255090:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655335808/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759495.255101:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655335680:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787759495.255108:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759495.255114:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759495.255116:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759495.255121:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12551 00000100:00080000:3.0:1787759495.255124:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759495.255128:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759495.255134:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759495.255152:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655335936/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759495.255164:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759495.255168:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759495.255170:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759495.255175:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12551 00000100:00080000:3.0:1787759495.255179:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787759495.255182:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787759495.255189:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759495.255194:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759495.255207:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655336064/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759495.255216:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759495.255219:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759495.255220:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759495.255225:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12551 00000100:00080000:3.0:1787759495.255228:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759495.255232:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759495.255237:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759495.255248:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655336192/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759495.255258:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759495.255261:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759495.255263:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759495.255282:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12551 00000100:00080000:3.0:1787759495.255286:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759495.255290:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759495.255295:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759495.255309:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655336320/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759495.255318:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759495.255322:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787759495.255324:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759495.255330:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12551 00000100:00080000:3.0:1787759495.255334:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759495.255347:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759495.255348:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759495/real 1787759495] req@ffff8c05a9f7aa00 x1874588655335680/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787759495.255360:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759495.255387:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655335680/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759495.255399:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759495.255405:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759495.255409:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655335680:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787759495.255439:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655335808:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759495.255450:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655336448/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787759495.255470:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759495.255485:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759495/real 1787759495] req@ffff8c05a9f78380 x1874588655335808/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759495.255498:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655335808/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759495.255507:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759495.255510:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759495.255514:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655335808:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759495.255521:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655335936:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759495.255542:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759495.255554:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759495/real 1787759495] req@ffff8c05a9f78a80 x1874588655335936/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759495.255613:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787759495.255614:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655335936/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759495.255627:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759495.255640:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759495.255641:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759495.255652:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787759495.255655:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12551 00000100:00100000:2.0:1787759495.255663:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655335936:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787759495.255666:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787759495.255676:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655336064:192.168.201.107@tcp:8:kworker.0 00000100:00080000:3.0:1787759495.255677:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759495.255695:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759495.255715:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655336576/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787759495.257030:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759495.257057:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759495/real 1787759495] req@ffff8c05a9f7a680 x1874588655336064/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759495.257073:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655336064/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759495.257082:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759495.257086:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759495.257090:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655336064:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759495.257098:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655336192:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759495.257129:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759495.257142:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759495/real 1787759495] req@ffff8c05a9f78700 x1874588655336192/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759495.257154:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655336192/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759495.257161:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759495.257164:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759495.257168:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655336192:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759495.257174:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655336320:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759495.257245:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759495.257259:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759495/real 1787759495] req@ffff8c05a9f7b480 x1874588655336320/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759495.257271:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655336320/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759495.257279:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759495.257282:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759495.257285:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655336320:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759495.257314:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655336448:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759495.257337:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759495.257347:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759495/real 1787759495] req@ffff8c05a9f79880 x1874588655336448/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759495.257358:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655336448/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759495.257366:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759495.257370:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759495.257385:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655336448:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759495.257392:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655336576:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759495.257413:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759495.257424:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759495/real 1787759495] req@ffff8c05a9f7a300 x1874588655336576/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759495.257435:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655336576/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759495.257443:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759495.257446:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759495.257449:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655336576:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759495.382028:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759495.830121:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759496.406069:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759496.854580:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759497.430028:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759497.878736:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759498.454053:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759498.902353:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759499.478382:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759499.926459:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787759500.374050:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759500.374061:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787759500.374067:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759500.374079:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12556 00000100:00080000:3.0:1787759500.374091:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759500.374100:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787759500.374149:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655336704/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759500.374239:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655336704:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787759500.374296:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759500.374300:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759500.374302:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759500.374307:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12556 00000100:00080000:3.0:1787759500.374311:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759500.374315:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759500.374326:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759500.374341:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655336832/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759500.374351:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759500.374354:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759500.374355:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759500.374359:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12556 00000100:00080000:3.0:1787759500.374362:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759500.374365:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759500.374370:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759500.374482:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655336960/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759500.374493:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759500.374496:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759500.374498:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759500.374501:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12556 00000100:00080000:3.0:1787759500.374505:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759500.374508:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759500.374513:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759500.374523:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655337088/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759500.374538:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759500.374541:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759500.374543:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759500.374546:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12556 00000100:00080000:3.0:1787759500.374549:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759500.374553:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759500.374558:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759500.374569:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655337216/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759500.374578:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759500.374581:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759500.374582:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759500.374586:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12556 00000100:00080000:3.0:1787759500.374589:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759500.374593:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759500.374597:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759500.374608:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655337344/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759500.374617:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759500.374621:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787759500.374622:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759500.374626:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12556 00000100:00080000:3.0:1787759500.374630:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759500.374633:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759500.374637:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759500.374646:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655337472/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759500.374654:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759500.374658:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759500.374659:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759500.374662:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12556 00000100:00080000:3.0:1787759500.374665:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759500.374668:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759500.374672:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759500.374682:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655337600/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787759500.374777:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759500.374919:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759500/real 1787759500] req@ffff8c05a9f7a300 x1874588655336704/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759500.374939:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655336704/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759500.374951:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759500.374955:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759500.374959:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655336704:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787759500.374984:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655336832:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759500.375009:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759500.375022:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759500/real 1787759500] req@ffff8c05a9f79880 x1874588655336832/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759500.375034:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655336832/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759500.375042:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759500.375045:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759500.375048:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655336832:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759500.375054:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655336960:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759500.375074:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759500.375085:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759500/real 1787759500] req@ffff8c05a9f7b480 x1874588655336960/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759500.375096:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655336960/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759500.375103:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759500.375106:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759500.375109:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655336960:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759500.375115:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655337088:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759500.375135:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759500.375145:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759500/real 1787759500] req@ffff8c05a9f78700 x1874588655337088/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759500.375156:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655337088/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759500.375164:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759500.375167:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759500.375170:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655337088:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759500.375175:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655337216:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759500.375195:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759500.375207:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759500/real 1787759500] req@ffff8c05a9f7a680 x1874588655337216/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759500.375218:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655337216/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759500.375225:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759500.375228:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759500.375231:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655337216:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759500.375237:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655337344:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759500.375255:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759500.375265:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759500/real 1787759500] req@ffff8c05a9f78a80 x1874588655337344/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759500.375300:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655337344/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759500.375308:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759500.375310:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759500.375313:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655337344:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759500.375319:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655337472:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759500.375338:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759500.375350:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759500/real 1787759500] req@ffff8c05a9f78380 x1874588655337472/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759500.375361:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655337472/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759500.375369:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759500.375383:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759500.375387:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655337472:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759500.375392:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655337600:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759500.375412:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759500.375422:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759500/real 1787759500] req@ffff8c05a9f7aa00 x1874588655337600/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759500.375433:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655337600/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759500.375446:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759500.375449:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759500.375453:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655337600:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759500.502151:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759500.950132:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759501.526286:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759501.975143:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759502.550085:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759502.998280:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759503.574291:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759504.022760:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759504.598828:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759505.046268:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787759505.494769:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759505.494781:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787759505.494787:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759505.494799:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12561 00000100:00080000:2.0:1787759505.494811:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759505.494820:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759505.494867:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588655337728/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759505.495814:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655337728:192.168.201.107@tcp:250:kworker.0 00000800:00000100:0.0:1787759505.495904:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:2.0:1787759505.495948:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759505.495952:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759505.495954:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759505.495959:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12561 00000100:00080000:2.0:1787759505.495964:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759505.495969:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759505.495979:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759505.495996:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588655337856/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759505.499147:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759505/real 1787759505] req@ffff8c05b2e72a00 x1874588655337728/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759505.499178:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588655337728/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759505.499201:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759505.499207:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759505.499212:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655337728:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787759505.499246:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655337856:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759505.499307:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759505.499327:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759505/real 1787759505] req@ffff8c05b2e70700 x1874588655337856/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759505.499340:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588655337856/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759505.499349:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759505.499353:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759505.499357:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655337856:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787759505.499687:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759505.499694:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759505.499697:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759505.499703:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12561 00000100:00080000:2.0:1787759505.499709:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759505.499717:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759505.499729:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759505.499757:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588655337984/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759505.499773:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759505.499776:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759505.499778:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759505.499781:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12561 00000100:00080000:2.0:1787759505.499784:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759505.499788:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759505.499792:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759505.499802:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588655338112/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759505.499811:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759505.499814:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759505.499816:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759505.499819:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12561 00000100:00080000:2.0:1787759505.499823:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759505.499826:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759505.499831:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759505.499841:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588655338240/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759505.499850:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759505.499853:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759505.499855:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759505.499859:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12561 00000100:00080000:2.0:1787759505.499862:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759505.499865:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759505.499869:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759505.499906:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588655338368/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759505.499916:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759505.499919:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787759505.499921:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759505.499925:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12561 00000100:00080000:2.0:1787759505.499928:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759505.499933:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759505.499937:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759505.499947:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588655338496/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759505.499963:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759505.499967:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759505.499969:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759505.499972:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12561 00000100:00080000:2.0:1787759505.499976:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759505.499980:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759505.499984:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759505.499995:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588655338624/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759505.500294:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655337984:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759505.500322:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759505.500337:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759505/real 1787759505] req@ffff8c05b2e73480 x1874588655337984/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759505.500350:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588655337984/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759505.500358:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759505.500362:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759505.500366:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655337984:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759505.500372:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655338112:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759505.500390:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759505.500400:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759505/real 1787759505] req@ffff8c05b2e71500 x1874588655338112/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759505.500409:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588655338112/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759505.500422:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759505.500425:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759505.500428:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655338112:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759505.500433:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655338240:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759505.500448:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759505.500457:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759505/real 1787759505] req@ffff8c05b2e73b80 x1874588655338240/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759505.500467:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588655338240/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759505.500474:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759505.500477:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759505.500480:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655338240:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759505.500485:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655338368:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759505.500504:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759505.500515:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759505/real 1787759505] req@ffff8c05b2e71c00 x1874588655338368/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759505.500526:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588655338368/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759505.500533:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759505.500536:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759505.500540:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655338368:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759505.500545:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655338496:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759505.500561:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759505.500571:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759505/real 1787759505] req@ffff8c05b2e71880 x1874588655338496/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759505.500581:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588655338496/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759505.500695:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759505.500699:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759505.500703:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655338496:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759505.500709:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655338624:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759505.500731:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759505.500743:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759505/real 1787759505] req@ffff8c05b2e70e00 x1874588655338624/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759505.500753:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588655338624/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759505.500759:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759505.500762:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759505.500766:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655338624:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759505.622129:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759506.070150:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759506.646004:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759507.094514:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759507.671655:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759508.121408:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759508.694676:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759509.142819:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759509.717998:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759510.166510:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787759510.614043:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759510.614063:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787759510.614072:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759510.614083:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12567 00000100:00080000:3.0:1787759510.614096:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759510.614114:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787759510.614222:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655338752/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759510.614254:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759510.614262:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759510.614264:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759510.614269:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12567 00000100:00080000:3.0:1787759510.614273:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759510.614279:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759510.614291:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759510.614305:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655338880/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759510.614311:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655338752:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787759510.614325:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759510.614329:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759510.614331:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759510.614335:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12567 00000100:00080000:3.0:1787759510.614340:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759510.614344:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759510.614349:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759510.614363:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655339008/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759510.614374:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759510.614379:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759510.614381:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759510.614385:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12567 00000100:00080000:3.0:1787759510.614389:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759510.614393:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759510.614399:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787759510.614406:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759510.614413:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655339136/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759510.614426:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759510.614430:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759510.614432:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759510.614436:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12567 00000100:00080000:3.0:1787759510.614439:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759510.614444:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759510.614449:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759510.614462:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655339264/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759510.614471:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759510.614475:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759510.614477:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787759510.614480:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759510/real 1787759510] req@ffff8c05a9f79500 x1874588655338752/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759510.614484:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12567 00000100:00080000:3.0:1787759510.614491:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759510.614498:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759510.614505:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759510.614509:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655338752/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759510.614520:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655339392/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759510.614524:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787759510.614532:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759510.614535:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787759510.614537:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759510.614544:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12567 00000100:00080000:0.0:1787759510.614546:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:3.0:1787759510.614548:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787759510.614554:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655338752:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787759510.614557:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759510.614566:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759510.614578:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655339520/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759510.614588:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655338880:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787759510.614589:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759510.614597:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759510.614599:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759510.614603:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12567 00000100:00080000:3.0:1787759510.614607:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759510.614611:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759510.614616:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787759510.614618:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759510.614632:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655339648/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759510.614633:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759510/real 1787759510] req@ffff8c05a9f7b800 x1874588655338880/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759510.614663:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655338880/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759510.614671:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759510.614674:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759510.614684:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655338880:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759510.614689:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655339008:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759510.614702:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759510.614709:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759510/real 1787759510] req@ffff8c05a9f78e00 x1874588655339008/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759510.614721:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655339008/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759510.614726:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759510.614728:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759510.614730:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655339008:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759510.614733:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655339136:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759510.614743:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759510.614757:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759510/real 1787759510] req@ffff8c05a9f7ad80 x1874588655339136/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759510.614771:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655339136/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759510.614779:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759510.614782:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759510.614799:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655339136:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759510.614804:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655339264:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759510.614824:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759510.614835:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759510/real 1787759510] req@ffff8c05a9f7b100 x1874588655339264/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759510.614846:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655339264/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759510.614861:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759510.614864:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759510.614866:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655339264:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759510.614870:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655339392:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759510.614905:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759510.614913:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759510/real 1787759510] req@ffff8c05a9f79f80 x1874588655339392/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759510.614921:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655339392/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759510.614926:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759510.614928:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759510.614930:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655339392:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759510.614980:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655339520:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759510.614999:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759510.615010:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759510/real 1787759510] req@ffff8c05a9f7bb80 x1874588655339520/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759510.615034:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655339520/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759510.615039:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759510.615041:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759510.615043:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655339520:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759510.615047:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655339648:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759510.615060:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759510.615077:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759510/real 1787759510] req@ffff8c05a9f79180 x1874588655339648/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759510.615086:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655339648/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759510.615093:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759510.615096:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759510.615099:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655339648:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759510.742145:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759511.190100:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759511.766331:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759512.214058:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759512.790002:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759513.238089:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759513.814220:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759514.264352:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759514.838150:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759515.286100:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787759515.733974:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759515.733984:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787759515.733989:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759515.736083:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12572 00000100:00080000:3.0:1787759515.736101:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759515.736113:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787759515.736171:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655339776/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759515.736206:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759515.736210:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759515.736213:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759515.736218:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12572 00000100:00080000:3.0:1787759515.736222:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759515.736226:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759515.736235:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759515.736248:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655339904/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759515.736258:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759515.736261:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759515.736263:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759515.736267:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12572 00000100:00080000:3.0:1787759515.736270:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759515.736274:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759515.736279:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759515.736289:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655340032/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759515.736298:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759515.736301:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759515.736303:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759515.736306:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12572 00000100:00080000:3.0:1787759515.736309:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759515.736312:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759515.736341:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759515.736377:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655340160/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759515.736385:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759515.736389:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759515.736390:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759515.736394:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12572 00000100:00080000:3.0:1787759515.736397:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759515.736401:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759515.736406:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759515.736415:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655340288/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759515.736437:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759515.736440:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759515.736442:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759515.736503:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12572 00000100:00080000:3.0:1787759515.736506:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759515.736510:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759515.736515:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759515.736526:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655340416/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759515.736535:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759515.736538:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787759515.736540:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759515.736543:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12572 00000100:00080000:3.0:1787759515.736546:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759515.736549:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759515.736553:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759515.736561:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655340544/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759515.736569:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759515.736572:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759515.736573:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759515.736576:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12572 00000100:00080000:3.0:1787759515.736578:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759515.736581:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759515.736673:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759515.736687:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655340672/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759515.736798:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655339776:192.168.201.107@tcp:250:kworker.0 00000800:00000100:0.0:1787759515.736872:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759515.736938:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759515/real 1787759515] req@ffff8c05a9f79180 x1874588655339776/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759515.736956:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655339776/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759515.736966:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759515.736970:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759515.736974:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655339776:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787759515.736982:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655339904:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759515.737004:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759515.737081:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759515/real 1787759515] req@ffff8c05a9f7bb80 x1874588655339904/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759515.737093:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655339904/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759515.737101:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759515.737104:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759515.737108:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655339904:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759515.737114:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655340032:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759515.737133:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759515.737144:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759515/real 1787759515] req@ffff8c05a9f79f80 x1874588655340032/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759515.737156:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655340032/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759515.737163:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759515.737167:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759515.737170:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655340032:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759515.737175:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655340160:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759515.737194:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759515.737204:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759515/real 1787759515] req@ffff8c05a9f7b100 x1874588655340160/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759515.737222:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655340160/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759515.737230:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759515.737233:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759515.737236:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655340160:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759515.737242:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655340288:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759515.737260:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759515.737271:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759515/real 1787759515] req@ffff8c05a9f7ad80 x1874588655340288/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759515.737282:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655340288/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759515.737289:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759515.737293:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759515.737296:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655340288:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759515.737302:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655340416:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759515.737320:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759515.737332:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759515/real 1787759515] req@ffff8c05a9f78e00 x1874588655340416/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759515.737342:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655340416/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759515.737354:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759515.737358:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759515.737361:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655340416:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759515.737366:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655340544:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759515.737383:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759515.737394:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759515/real 1787759515] req@ffff8c05a9f7b800 x1874588655340544/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759515.737403:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655340544/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759515.737411:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759515.737414:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759515.737417:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655340544:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759515.737423:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655340672:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759515.737440:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759515.737451:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759515/real 1787759515] req@ffff8c05a9f79500 x1874588655340672/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759515.737461:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655340672/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759515.737469:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759515.737472:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759515.737476:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655340672:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759515.862936:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759516.310480:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759516.885983:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759517.334473:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759517.910011:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759518.358183:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759518.933945:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759519.382222:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759519.958004:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759520.406163:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787759520.854090:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759520.854102:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787759520.854106:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759520.854116:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12577 00000100:00080000:0.0:1787759520.854129:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759520.854136:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787759520.854179:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655340800/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759520.854700:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655340800:192.168.201.107@tcp:250:kworker.0 00000800:00000100:2.0:1787759520.854757:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787759520.854790:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759520.854792:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787759520.854793:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759520.854797:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12577 00000100:00080000:0.0:1787759520.854800:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759520.854803:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759520.854811:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759520.854822:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588655340928/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759520.855026:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759520/real 1787759520] req@ffff8c05a9f75880 x1874588655340800/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759520.855047:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655340800/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759520.855063:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759520.855066:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759520.855070:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655340800:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787759520.855094:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655340928:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759520.855117:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759520.855130:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759520/real 1787759520] req@ffff8c05a9f75500 x1874588655340928/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759520.855143:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588655340928/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759520.855151:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759520.855154:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759520.855157:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655340928:192.168.201.107@tcp:38:kworker.0 00000100:00080000:0.0:1787759520.855450:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759520.855454:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787759520.855455:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759520.855458:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12577 00000100:00080000:0.0:1787759520.855462:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759520.855465:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759520.855470:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759520.855480:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74000 x1874588655341056/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759520.855487:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759520.855490:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787759520.855491:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759520.855493:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12577 00000100:00080000:0.0:1787759520.855495:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759520.855497:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759520.855500:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759520.855505:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75180 x1874588655341184/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759520.855510:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759520.855512:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787759520.855513:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759520.855515:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12577 00000100:00080000:0.0:1787759520.855516:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759520.855518:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759520.855520:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759520.855525:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74e00 x1874588655341312/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759520.855535:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759520.855538:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787759520.855540:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759520.855543:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12577 00000100:00080000:0.0:1787759520.855546:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759520.855548:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759520.855552:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759520.855562:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77100 x1874588655341440/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759520.855570:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759520.855573:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787759520.855575:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759520.855578:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12577 00000100:00080000:0.0:1787759520.855581:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759520.855585:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759520.855589:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759520.855598:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76d80 x1874588655341568/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759520.855606:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759520.855609:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787759520.855611:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759520.855614:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12577 00000100:00080000:0.0:1787759520.855618:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759520.856295:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759520.856299:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759520.856310:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75500 x1874588655341696/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759520.856366:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655341056:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759520.856393:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759520.856407:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759520/real 1787759520] req@ffff8c05a9f74000 x1874588655341056/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759520.856420:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74000 x1874588655341056/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759520.856428:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759520.856430:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759520.856433:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655341056:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759520.856437:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655341184:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759520.856453:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759520.856460:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759520/real 1787759520] req@ffff8c05a9f75180 x1874588655341184/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759520.856467:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75180 x1874588655341184/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759520.856471:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759520.856473:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759520.856475:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655341184:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759520.856478:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655341312:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759520.856487:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759520.856492:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759520/real 1787759520] req@ffff8c05a9f74e00 x1874588655341312/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759520.856498:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74e00 x1874588655341312/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759520.856503:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759520.856504:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759520.856506:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655341312:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759520.856509:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655341440:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759520.856518:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759520.856523:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759520/real 1787759520] req@ffff8c05a9f77100 x1874588655341440/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759520.856530:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77100 x1874588655341440/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759520.856536:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759520.856538:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759520.856541:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655341440:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759520.856546:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655341568:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759520.856563:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759520.856573:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759520/real 1787759520] req@ffff8c05a9f76d80 x1874588655341568/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759520.856585:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76d80 x1874588655341568/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759520.856592:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759520.856595:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759520.856598:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655341568:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759520.856603:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655341696:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759520.856685:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759520.856697:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759520/real 1787759520] req@ffff8c05a9f75500 x1874588655341696/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759520.856704:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75500 x1874588655341696/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759520.856712:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759520.856714:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759520.856716:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655341696:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759520.982282:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759521.430256:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759522.005990:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759522.454085:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759523.030031:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759523.478150:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759524.054006:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759524.502044:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759525.077975:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759525.526116:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787759525.973993:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759525.974003:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787759525.974006:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759525.974014:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12582 00000100:00080000:3.0:1787759525.974023:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759525.974029:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787759525.974060:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655341824/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759525.974083:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759525.974085:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759525.974086:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759525.974089:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12582 00000100:00080000:3.0:1787759525.974091:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759525.974093:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759525.974100:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759525.974107:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655341952/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759525.974114:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759525.974118:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759525.974119:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759525.974123:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12582 00000100:00080000:3.0:1787759525.974126:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759525.974130:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759525.974133:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759525.974139:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655342080/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759525.974145:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759525.974146:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759525.974147:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759525.974150:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12582 00000100:00080000:3.0:1787759525.974151:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759525.974154:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759525.974157:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759525.974163:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655342208/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759525.974178:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655341824:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787759525.974178:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759525.974192:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759525.974194:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759525.974198:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12582 00000100:00080000:3.0:1787759525.974200:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759525.974202:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759525.974207:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759525.974220:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655342336/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759525.974225:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759525.974227:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759525.974228:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759525.974230:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12582 00000100:00080000:3.0:1787759525.974233:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759525.974235:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759525.974239:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759525.974245:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655342464/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759525.974250:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759525.974253:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787759525.974254:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759525.974256:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12582 00000100:00080000:3.0:1787759525.974259:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759525.974260:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759525.974264:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787759525.974269:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759525.974279:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655342592/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759525.974285:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759525.974288:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759525.974290:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759525.974292:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12582 00000100:00080000:3.0:1787759525.974294:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759525.974298:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759525.974302:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759525.974309:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655342720/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759525.974338:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759525/real 1787759525] req@ffff8c05a9f7aa00 x1874588655341824/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759525.974357:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655341824/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759525.974374:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759525.974377:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759525.974380:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655341824:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787759525.974386:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655341952:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759525.974405:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759525.974413:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759525/real 1787759525] req@ffff8c05a9f78380 x1874588655341952/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759525.974421:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655341952/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759525.974426:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759525.974429:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759525.974431:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655341952:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759525.974434:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655342080:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759525.974444:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759525.974449:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759525/real 1787759525] req@ffff8c05a9f78a80 x1874588655342080/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759525.974480:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655342080/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759525.974485:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759525.974487:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759525.974489:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655342080:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759525.974509:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655342208:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759525.974523:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759525.974529:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759525/real 1787759525] req@ffff8c05a9f7a680 x1874588655342208/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759525.974536:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655342208/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759525.974541:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759525.974543:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759525.974546:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655342208:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759525.974550:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655342336:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759525.974560:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759525.974565:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759525/real 1787759525] req@ffff8c05a9f78700 x1874588655342336/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759525.974571:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655342336/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759525.974576:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759525.974577:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759525.974579:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655342336:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759525.974582:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655342464:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759525.974591:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759525.974596:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759525/real 1787759525] req@ffff8c05a9f7b480 x1874588655342464/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759525.974602:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655342464/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759525.974607:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759525.974608:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759525.974610:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655342464:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759525.974613:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655342592:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759525.974621:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759525.974627:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759525/real 1787759525] req@ffff8c05a9f79880 x1874588655342592/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759525.974633:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655342592/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759525.974638:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759525.974639:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759525.974641:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655342592:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759525.974644:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655342720:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759525.974652:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759525.974657:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759525/real 1787759525] req@ffff8c05a9f7a300 x1874588655342720/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759525.974663:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655342720/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759525.974667:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759525.974668:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759525.974670:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655342720:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759526.101961:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759526.550053:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759527.127903:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759527.574112:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759528.149982:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759528.598080:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759529.173959:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759529.622082:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759530.198033:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759530.646093:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787759531.094315:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759531.094326:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787759531.094331:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759531.094343:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12587 00000100:00080000:3.0:1787759531.094355:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759531.094365:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787759531.094482:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655342848/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759531.094568:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655342848:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787759531.094820:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759531.094825:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759531.094827:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759531.094833:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12587 00000100:00080000:3.0:1787759531.094838:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759531.094843:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759531.094860:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759531.096207:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655342976/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759531.096239:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759531.096244:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759531.096246:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759531.096252:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12587 00000100:00080000:3.0:1787759531.096260:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759531.096267:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759531.096279:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759531.096299:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655343104/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759531.096311:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759531.096317:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787759531.096318:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655342976:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787759531.096319:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759531.096328:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12587 00000100:00080000:3.0:1787759531.096333:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759531.096337:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759531.096343:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759531.096357:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655343232/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759531.096367:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759531.096371:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759531.096373:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759531.096378:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12587 00000100:00080000:3.0:1787759531.096382:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759531.096386:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759531.096391:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759531.096403:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655343104:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759531.096405:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655343360/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759531.096426:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655343232:192.168.201.107@tcp:8:kworker.0 00000100:00080000:3.0:1787759531.099980:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759531.099987:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759531.099991:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759531.100002:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12587 00000100:00080000:3.0:1787759531.100014:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787759531.100021:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655343360:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787759531.100024:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759531.100038:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759531.100076:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655343488/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759531.100088:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759531.100092:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787759531.100093:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759531.100097:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12587 00000100:00080000:3.0:1787759531.100101:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759531.100106:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759531.100111:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759531.100123:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655343616/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759531.100134:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759531.100138:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759531.100140:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759531.100144:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12587 00000100:00080000:3.0:1787759531.100147:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787759531.100150:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655343488:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787759531.100155:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759531.100163:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759531.100178:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655343744/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759531.100186:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655343616:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759531.100217:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655343744:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759531.222013:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759531.670124:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759532.246025:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759532.694048:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759533.269988:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759533.718099:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787759534.166183:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.107:988 00000400:00000100:1.0:1787759534.166224:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.107@tcp at host 192.168.201.107 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787759534.166239:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787759534.166284:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787759534.166293:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787759534.166314:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787759534.166325:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787759534.166336:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787759534.166344:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000800:00000100:1.0:1787759534.166353:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.107@tcp 00000100:00100000:0.0:1787759534.166450:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759531/real 1787759534] req@ffff8c05a9f7aa00 x1874588655343744/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759534.166476:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655343744/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759534.166495:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759534.166500:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759534.166505:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655343744:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759534.166512:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759531/real 1787759534] req@ffff8c05a9f78a80 x1874588655343488/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759534.166526:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655343488/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759534.166534:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759534.166538:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759534.166541:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655343488:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759534.166548:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759531/real 1787759534] req@ffff8c05a9f78380 x1874588655343616/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759534.166560:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655343616/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759534.166567:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759534.166571:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759534.166574:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655343616:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759534.166581:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759531/real 1787759534] req@ffff8c05a9f7a680 x1874588655343360/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759534.166592:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655343360/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759534.166605:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759534.166609:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759534.166613:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655343360:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759534.166619:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759531/real 1787759534] req@ffff8c05a9f7b480 x1874588655343104/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759534.166632:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655343104/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759534.166639:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759534.166642:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759534.166645:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655343104:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759534.166652:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759531/real 1787759534] req@ffff8c05a9f78700 x1874588655343232/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759534.166668:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655343232/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759534.166676:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759534.166679:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759534.166682:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655343232:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759534.166702:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759531/real 1787759534] req@ffff8c05a9f79880 x1874588655342976/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759534.166714:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655342976/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759534.166721:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759534.166724:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759534.166728:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655342976:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759534.166734:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759531/real 1787759534] req@ffff8c05a9f7a300 x1874588655342848/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759534.166746:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655342848/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759534.166752:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759534.166755:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759534.166758:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655342848:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787759534.293987:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759534.742081:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787759535.318027:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759535.766164:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787759536.214087:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759536.214097:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787759536.214100:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759536.214110:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12592 00000100:00080000:1.0:1787759536.214120:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759536.214126:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759536.214168:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655343872/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759536.214205:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759536.214209:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759536.214211:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759536.214215:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12592 00000100:00080000:1.0:1787759536.214219:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759536.214223:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759536.214233:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759536.214246:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655344000/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759536.214255:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759536.214258:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759536.214259:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759536.214262:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12592 00000100:00080000:1.0:1787759536.214265:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759536.214269:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759536.214273:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759536.214284:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655344128/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759536.214294:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759536.214297:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759536.214298:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759536.214301:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12592 00000100:00080000:1.0:1787759536.214304:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759536.214308:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759536.214311:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759536.214320:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655344256/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759536.214328:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759536.214331:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759536.214333:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759536.214336:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12592 00000100:00080000:1.0:1787759536.214338:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759536.214342:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759536.214346:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759536.214355:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655344384/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759536.214363:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759536.214366:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759536.214367:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759536.214377:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12592 00000100:00080000:1.0:1787759536.214381:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759536.214384:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759536.214389:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759536.214399:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655344512/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759536.214407:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759536.214409:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787759536.214411:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759536.214414:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12592 00000100:00080000:1.0:1787759536.214429:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759536.214434:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759536.214438:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759536.214447:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655344640/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759536.214455:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759536.214457:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759536.214458:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759536.214462:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12592 00000100:00080000:1.0:1787759536.214465:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759536.214468:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759536.214472:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759536.214480:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655344768/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759536.215624:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655343872:192.168.201.107@tcp:250:kworker.0 00000800:00000100:0.0:1787759536.215728:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759536.215777:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759536/real 1787759536] req@ffff8c05b2d99880 x1874588655343872/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759536.215797:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655343872/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759536.215809:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759536.215813:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759536.215818:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655343872:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787759536.215826:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655344000:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759536.215850:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759536.215862:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759536/real 1787759536] req@ffff8c05b2d98000 x1874588655344000/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759536.215873:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655344000/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759536.215911:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759536.215915:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759536.215918:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655344000:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759536.215925:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655344128:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759536.215945:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759536.215956:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759536/real 1787759536] req@ffff8c05b2d98700 x1874588655344128/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759536.215968:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655344128/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759536.215975:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759536.215978:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759536.215981:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655344128:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759536.215987:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655344256:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759536.216008:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759536.216019:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759536/real 1787759536] req@ffff8c05b2d99f80 x1874588655344256/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759536.216030:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655344256/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759536.216038:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759536.216041:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759536.216046:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655344256:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759536.216053:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655344384:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759536.216073:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759536.216084:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759536/real 1787759536] req@ffff8c05b2d9bb80 x1874588655344384/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759536.216094:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655344384/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759536.216113:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759536.216117:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759536.216120:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655344384:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759536.216126:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655344512:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759536.216145:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759536.216155:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759536/real 1787759536] req@ffff8c05b2d9b100 x1874588655344512/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759536.216166:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655344512/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759536.216174:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759536.216177:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759536.216180:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655344512:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759536.216186:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655344640:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759536.216207:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759536.216218:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759536/real 1787759536] req@ffff8c05b2d9b480 x1874588655344640/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759536.216229:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655344640/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759536.216237:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759536.216240:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759536.216243:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655344640:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759536.216256:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655344768:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759536.216275:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759536.216285:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759536/real 1787759536] req@ffff8c05b2d9aa00 x1874588655344768/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759536.216297:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655344768/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759536.216305:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759536.216308:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759536.216311:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655344768:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759536.342059:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759536.793149:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759537.366038:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759537.814690:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759538.389996:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759538.838335:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759539.413982:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759539.866025:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759540.438224:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759540.886128:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787759541.333974:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759541.333985:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787759541.333990:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759541.334007:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12597 00000100:00080000:3.0:1787759541.334019:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759541.334027:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787759541.334067:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655344896/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759541.334096:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759541.334099:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759541.334101:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759541.334106:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12597 00000100:00080000:3.0:1787759541.334109:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759541.334114:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759541.334124:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759541.334135:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655344896:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787759541.334139:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655345024/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759541.334151:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759541.334155:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759541.334157:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759541.334161:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12597 00000100:00080000:3.0:1787759541.334165:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759541.334169:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759541.334173:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759541.334184:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655345152/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759541.334194:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759541.334198:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759541.334200:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787759541.334203:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787759541.334218:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12597 00000100:00080000:3.0:1787759541.334223:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759541.334227:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759541.334233:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759541.334242:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759541/real 1787759541] req@ffff8c05a9f79500 x1874588655344896/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787759541.334245:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655345280/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759541.334265:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787759541.334271:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655344896/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759541.334275:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759541.334277:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759541.334282:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12597 00000100:00080000:3.0:1787759541.334286:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759541.334293:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787759541.334299:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787759541.334299:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787759541.334307:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759541.334313:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655344896:192.168.201.107@tcp:250:kworker.0 00000100:00100000:3.0:1787759541.334315:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655345408/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759541.334330:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759541.334334:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759541.334336:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787759541.334339:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655345024:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787759541.334343:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12597 00000100:00080000:3.0:1787759541.334348:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759541.334352:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759541.334357:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787759541.334367:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:3.0:1787759541.334371:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655345536/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759541.334385:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759541/real 1787759541] req@ffff8c05a9f7b800 x1874588655345024/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759541.334387:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759541.334395:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787759541.334399:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759541.334406:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12597 00000100:00080000:3.0:1787759541.334409:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787759541.334410:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655345024/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759541.334417:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787759541.334421:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787759541.334424:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787759541.334426:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759541.334430:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655345024:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759541.334436:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655345152:192.168.201.107@tcp:38:kworker.0 00000100:00100000:3.0:1787759541.334439:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655345664/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759541.334448:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759541.334451:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759541.334453:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787759541.334456:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:3.0:1787759541.334460:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12597 00000100:00080000:3.0:1787759541.334465:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787759541.334468:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759541/real 1787759541] req@ffff8c05a9f78e00 x1874588655345152/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787759541.334472:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759541.334481:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759541.334484:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655345152/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759541.334491:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759541.334496:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:3.0:1787759541.334496:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655345792/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759541.334502:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655345152:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759541.334512:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655345280:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759541.334528:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759541.334540:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759541/real 1787759541] req@ffff8c05a9f7ad80 x1874588655345280/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759541.334551:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655345280/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759541.334559:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759541.334573:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759541.334576:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655345280:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759541.334582:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655345408:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759541.334606:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759541.334619:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759541/real 1787759541] req@ffff8c05a9f7b100 x1874588655345408/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759541.334627:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655345408/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759541.334632:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759541.334635:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759541.334636:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655345408:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759541.334648:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655345536:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759541.334659:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759541.334665:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759541/real 1787759541] req@ffff8c05a9f79f80 x1874588655345536/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759541.334673:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655345536/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759541.334681:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759541.334684:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759541.334687:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655345536:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759541.334692:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655345664:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759541.334710:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759541.334721:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759541/real 1787759541] req@ffff8c05a9f7bb80 x1874588655345664/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759541.334729:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655345664/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759541.334734:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759541.334736:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759541.334738:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655345664:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759541.334742:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655345792:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759541.334754:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759541.334760:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759541/real 1787759541] req@ffff8c05a9f79180 x1874588655345792/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759541.334766:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655345792/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759541.334775:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759541.334778:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759541.334779:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655345792:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759541.461979:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759541.910106:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759542.486017:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759542.934336:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759543.510034:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759543.958140:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759544.534047:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759544.982267:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759545.557980:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759546.006190:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787759546.454099:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759546.454111:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787759546.454116:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759546.454127:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12602 00000100:00080000:2.0:1787759546.454139:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759546.454147:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759546.454193:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73800 x1874588655345920/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759546.454228:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759546.454231:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759546.454233:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759546.454246:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12602 00000100:00080000:2.0:1787759546.454251:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759546.454255:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759546.454266:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759546.454301:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70e00 x1874588655346048/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759546.454317:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759546.454320:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759546.454321:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759546.454324:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12602 00000100:00080000:2.0:1787759546.454327:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759546.454330:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759546.454334:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759546.454346:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71880 x1874588655346176/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759546.454355:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759546.454359:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759546.454360:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759546.454364:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12602 00000100:00080000:2.0:1787759546.454368:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759546.454372:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759546.454376:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759546.454387:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71c00 x1874588655346304/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759546.454396:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759546.454399:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759546.454414:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759546.454418:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12602 00000100:00080000:2.0:1787759546.454423:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759546.454427:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759546.454432:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759546.454443:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73b80 x1874588655346432/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759546.454452:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759546.454455:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759546.454457:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759546.454460:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12602 00000100:00080000:2.0:1787759546.454463:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759546.454467:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759546.454481:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759546.454493:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71500 x1874588655346560/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759546.454502:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759546.454505:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787759546.454507:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759546.454510:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12602 00000100:00080000:2.0:1787759546.454513:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759546.454517:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759546.454521:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759546.454531:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e73480 x1874588655346688/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759546.454545:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759546.454548:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759546.454550:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759546.454554:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12602 00000100:00080000:2.0:1787759546.454557:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759546.454560:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759546.454565:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787759546.454573:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70700 x1874588655346816/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759546.454682:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655345920:192.168.201.107@tcp:250:kworker.0 00000800:00000100:0.0:1787759546.454759:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759546.454802:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759546/real 1787759546] req@ffff8c05b2e73800 x1874588655345920/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759546.454820:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73800 x1874588655345920/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759546.454830:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759546.454834:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759546.454838:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655345920:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787759546.454845:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655346048:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759546.454868:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759546.454902:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759546/real 1787759546] req@ffff8c05b2e70e00 x1874588655346048/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759546.454915:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70e00 x1874588655346048/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759546.454928:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759546.454931:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759546.454934:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655346048:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759546.454940:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655346176:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759546.454974:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759546.454985:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759546/real 1787759546] req@ffff8c05b2e71880 x1874588655346176/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759546.454995:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71880 x1874588655346176/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759546.455002:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759546.455005:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759546.455008:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655346176:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759546.455013:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655346304:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759546.455032:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759546.455043:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759546/real 1787759546] req@ffff8c05b2e71c00 x1874588655346304/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759546.455052:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71c00 x1874588655346304/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759546.455059:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759546.455061:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759546.455065:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655346304:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759546.455071:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655346432:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759546.455089:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759546.455099:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759546/real 1787759546] req@ffff8c05b2e73b80 x1874588655346432/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759546.455110:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73b80 x1874588655346432/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759546.455118:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759546.455121:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759546.455124:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655346432:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759546.455129:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655346560:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759546.455145:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759546.455155:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759546/real 1787759546] req@ffff8c05b2e71500 x1874588655346560/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759546.455166:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71500 x1874588655346560/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759546.455173:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759546.455176:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759546.455179:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655346560:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759546.455184:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655346688:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759546.455201:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759546.455209:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759546/real 1787759546] req@ffff8c05b2e73480 x1874588655346688/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759546.455216:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e73480 x1874588655346688/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759546.455221:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759546.455223:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759546.455225:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655346688:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759546.455228:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655346816:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759546.455237:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759546.455242:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759546/real 1787759546] req@ffff8c05b2e70700 x1874588655346816/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759546.455248:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70700 x1874588655346816/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759546.455253:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759546.455254:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759546.455256:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655346816:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759546.582548:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759547.031001:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759547.606553:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759548.054785:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759548.630562:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759549.078168:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759549.653997:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759550.102136:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759550.679261:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759551.126449:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787759551.573990:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759551.574000:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787759551.574005:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759551.574016:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12607 00000100:00080000:2.0:1787759551.574027:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759551.574034:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:2.0:1787759551.574073:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72a00 x1874588655346944/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759551.574109:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759551.574112:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759551.574114:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759551.574118:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12607 00000100:00080000:2.0:1787759551.574121:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759551.574125:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759551.574136:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759551.574153:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72d80 x1874588655347072/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759551.574164:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655346944:192.168.201.107@tcp:250:kworker.0 00000100:00080000:2.0:1787759551.574171:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759551.574176:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759551.574178:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759551.574182:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12607 00000100:00080000:2.0:1787759551.574186:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759551.574189:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759551.574194:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759551.574205:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e71180 x1874588655347200/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759551.574214:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759551.574218:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759551.574220:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759551.574223:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12607 00000100:00080000:2.0:1787759551.574231:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759551.574238:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759551.574242:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787759551.574247:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759551.574256:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72680 x1874588655347328/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759551.574266:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759551.574269:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787759551.574272:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759551.574275:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12607 00000100:00080000:2.0:1787759551.574278:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759551.574282:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759551.574287:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759551.574296:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70380 x1874588655347456/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759551.574301:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759551/real 1787759551] req@ffff8c05b2e72a00 x1874588655346944/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759551.574316:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759551.574320:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787759551.574322:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759551.574326:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12607 00000100:00100000:0.0:1787759551.574331:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72a00 x1874588655346944/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759551.574332:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759551.574339:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00080000:0.0:1787759551.574350:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787759551.574352:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787759551.574357:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759551.574362:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655346944:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787759551.574366:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70000 x1874588655347584/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759551.574377:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759551.574394:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787759551.574396:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787759551.574399:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655347072:192.168.201.107@tcp:38:kworker.0 00000100:00080000:2.0:1787759551.574403:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12607 00000100:00080000:2.0:1787759551.574408:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787759551.574412:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759551.574417:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787759551.574428:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759551.574432:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e70a80 x1874588655347712/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759551.574444:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759551/real 1787759551] req@ffff8c05b2e72d80 x1874588655347072/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759551.574444:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787759551.574452:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787759551.574455:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787759551.574463:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12607 00000100:00080000:2.0:1787759551.574468:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787759551.574469:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72d80 x1874588655347072/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759551.574474:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:2.0:1787759551.574483:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787759551.574485:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759551.574491:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759551.574494:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655347072:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759551.574501:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655347200:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759551.574505:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2e72300 x1874588655347840/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787759551.574523:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759551.574535:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759551/real 1787759551] req@ffff8c05b2e71180 x1874588655347200/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759551.574546:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e71180 x1874588655347200/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759551.574554:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759551.574558:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759551.574560:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655347200:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759551.574566:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655347328:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759551.574581:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759551.574589:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759551/real 1787759551] req@ffff8c05b2e72680 x1874588655347328/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759551.574600:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72680 x1874588655347328/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759551.574607:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759551.574610:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759551.574613:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655347328:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759551.574618:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655347456:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759551.574631:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759551.574639:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759551/real 1787759551] req@ffff8c05b2e70380 x1874588655347456/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759551.574650:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70380 x1874588655347456/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759551.574658:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759551.574660:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759551.574663:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655347456:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759551.574668:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655347584:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759551.574681:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759551.574689:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759551/real 1787759551] req@ffff8c05b2e70000 x1874588655347584/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759551.574700:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70000 x1874588655347584/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759551.574708:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759551.574710:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759551.574713:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655347584:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759551.574739:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655347712:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759551.574754:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759551.574763:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759551/real 1787759551] req@ffff8c05b2e70a80 x1874588655347712/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759551.574775:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e70a80 x1874588655347712/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759551.574782:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759551.574785:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759551.574789:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655347712:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759551.574794:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655347840:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759551.574807:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759551.574815:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759551/real 1787759551] req@ffff8c05b2e72300 x1874588655347840/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759551.574825:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2e72300 x1874588655347840/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759551.574839:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759551.574842:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759551.574845:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2e72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655347840:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759551.701980:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759552.150080:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759552.725972:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759553.174060:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759553.750000:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759554.198202:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759554.773970:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759555.222054:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759555.798011:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759556.246105:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787759556.694028:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759556.694041:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787759556.694046:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759556.694058:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12613 00000100:00080000:3.0:1787759556.694070:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759556.694079:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787759556.694128:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655347968/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759556.694165:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759556.694169:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759556.694170:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759556.694175:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12613 00000100:00080000:3.0:1787759556.694179:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759556.694189:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759556.694201:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759556.694217:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655348096/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759556.694232:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759556.694235:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759556.694237:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759556.694241:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12613 00000100:00080000:3.0:1787759556.694245:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759556.694248:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759556.694253:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759556.694264:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655348224/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759556.694274:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759556.694278:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759556.694280:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759556.694283:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12613 00000100:00080000:3.0:1787759556.694286:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759556.694310:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759556.694316:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759556.694328:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655348352/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759556.694338:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759556.694349:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759556.694351:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759556.694355:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12613 00000100:00080000:3.0:1787759556.694359:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759556.694365:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787759556.694367:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655347968:192.168.201.107@tcp:250:kworker.0 00000008:00100000:3.0:1787759556.694371:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759556.694386:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655348480/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759556.694396:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759556.694399:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759556.694401:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759556.694405:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12613 00000100:00080000:3.0:1787759556.694409:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759556.694412:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759556.694418:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759556.694429:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655348608/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759556.694437:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759556.694441:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787759556.694443:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759556.694447:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12613 00000100:00080000:3.0:1787759556.694451:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759556.694455:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:0.0:1787759556.694461:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:3.0:1787759556.694464:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759556.694479:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655348736/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759556.694489:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759556.694492:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759556.694494:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759556.694499:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12613 00000100:00080000:3.0:1787759556.694502:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759556.694506:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759556.694511:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759556.694522:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655348864/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759556.694535:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759556/real 1787759556] req@ffff8c05a9f79180 x1874588655347968/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759556.694556:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655347968/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759556.694567:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759556.694571:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759556.694575:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655347968:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787759556.694583:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655348096:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759556.694604:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759556.694615:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759556/real 1787759556] req@ffff8c05a9f7bb80 x1874588655348096/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759556.694626:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655348096/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759556.694633:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759556.694636:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759556.694639:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655348096:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759556.694645:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655348224:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759556.694660:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759556.694670:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759556/real 1787759556] req@ffff8c05a9f79f80 x1874588655348224/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759556.694680:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655348224/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759556.694687:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759556.694690:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759556.694693:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655348224:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759556.694698:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655348352:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759556.694714:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759556.694746:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759556/real 1787759556] req@ffff8c05a9f7b100 x1874588655348352/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759556.694757:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655348352/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759556.694765:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759556.694768:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759556.694771:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655348352:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759556.694798:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655348480:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759556.694818:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759556.694829:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759556/real 1787759556] req@ffff8c05a9f7ad80 x1874588655348480/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759556.694841:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655348480/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759556.694849:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759556.694852:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759556.694855:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655348480:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759556.694860:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655348608:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759556.694910:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759556.694922:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759556/real 1787759556] req@ffff8c05a9f78e00 x1874588655348608/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759556.694933:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655348608/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759556.694940:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759556.694943:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759556.694946:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655348608:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759556.694951:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655348736:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759556.694967:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759556.694975:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759556/real 1787759556] req@ffff8c05a9f7b800 x1874588655348736/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759556.694990:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655348736/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759556.694997:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759556.695000:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759556.695003:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655348736:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759556.695008:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655348864:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759556.695023:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759556.695031:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759556/real 1787759556] req@ffff8c05a9f79500 x1874588655348864/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759556.695041:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655348864/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759556.695047:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759556.695050:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759556.695052:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655348864:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759556.822010:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759557.270076:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759557.846017:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759558.294144:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759558.869975:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759559.318032:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759559.893993:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759560.342106:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759560.918003:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759561.366273:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787759561.814063:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759561.814076:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787759561.814080:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759561.814096:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12618 00000100:00080000:1.0:1787759561.814110:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759561.814118:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759561.814164:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588655348992/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759561.814196:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759561.814215:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759561.814218:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759561.814222:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12618 00000100:00080000:1.0:1787759561.814226:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759561.814230:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759561.814239:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759561.814255:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655349120/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759561.814266:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787759561.814273:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655348992:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787759561.814273:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759561.814276:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759561.814285:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12618 00000100:00080000:1.0:1787759561.814288:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759561.814292:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759561.814297:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759561.814310:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655349248/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759561.814321:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759561.814324:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759561.814325:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759561.814329:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12618 00000100:00080000:1.0:1787759561.814333:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759561.814337:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:0.0:1787759561.814342:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:1.0:1787759561.814343:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759561.814356:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655349376/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759561.814366:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759561.814375:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759561.814377:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759561.814381:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12618 00000100:00080000:1.0:1787759561.814385:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759561.814389:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787759561.814394:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759561/real 1787759561] req@ffff8c05b2d9a300 x1874588655348992/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787759561.814396:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759561.814411:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655349504/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759561.814418:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588655348992/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787759561.814424:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759561.814432:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759561.814435:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759561.814438:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787759561.814441:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12618 00000100:00080000:0.0:1787759561.814446:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:1.0:1787759561.814447:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787759561.814453:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655348992:192.168.201.107@tcp:250:kworker.0 00000100:00080000:1.0:1787759561.814454:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759561.814463:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759561.814476:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655349632/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759561.814485:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655349120:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787759561.814485:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759561.814493:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787759561.814495:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787759561.814513:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759561.814526:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759561/real 1787759561] req@ffff8c05b2d98380 x1874588655349120/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759561.814537:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655349120/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759561.814545:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759561.814549:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759561.814552:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655349120:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759561.814558:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655349248:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787759561.814566:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12618 00000100:00080000:1.0:1787759561.814574:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787759561.814578:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787759561.814582:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759561.814588:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759561.814591:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759561/real 1787759561] req@ffff8c05b2d9aa00 x1874588655349248/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759561.814603:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655349248/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759561.814612:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787759561.814614:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655349760/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759561.814618:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759561.814625:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655349248:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787759561.814631:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787759561.814636:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655349376:192.168.201.107@tcp:8:kworker.0 00000100:00080000:1.0:1787759561.814637:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759561.814641:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759561.814646:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12618 00000100:00080000:1.0:1787759561.814657:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787759561.814661:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787759561.814663:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759561.814670:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759561.814677:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759561/real 1787759561] req@ffff8c05b2d9b480 x1874588655349376/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787759561.814682:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655349888/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759561.814701:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655349376/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759561.814711:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759561.814727:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759561.814731:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655349376:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759561.814737:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655349504:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759561.814756:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759561.814766:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759561/real 1787759561] req@ffff8c05b2d9b100 x1874588655349504/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759561.814777:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655349504/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759561.814785:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759561.814789:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759561.814792:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655349504:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759561.814810:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655349632:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759561.814828:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759561.814838:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759561/real 1787759561] req@ffff8c05b2d9bb80 x1874588655349632/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759561.814848:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655349632/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759561.814856:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759561.814859:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759561.814862:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655349632:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759561.814867:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655349760:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759561.814915:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759561.814925:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759561/real 1787759561] req@ffff8c05b2d99f80 x1874588655349760/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759561.814936:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655349760/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759561.814943:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759561.814946:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759561.814949:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655349760:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759561.814954:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655349888:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759561.814970:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759561.814980:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759561/real 1787759561] req@ffff8c05b2d98700 x1874588655349888/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759561.814990:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655349888/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759561.815004:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759561.815019:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759561.815022:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655349888:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759561.942023:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759562.393729:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759562.966022:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759563.414131:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759563.990013:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759564.438069:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759565.014012:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759565.464855:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759566.038046:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759566.486116:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787759566.934147:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759566.934156:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787759566.934160:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759566.934171:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12623 00000100:00080000:1.0:1787759566.934184:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759566.934192:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759566.934226:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655350016/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759566.934299:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655350016:192.168.201.107@tcp:250:kworker.0 00000800:00000100:0.0:1787759566.934360:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787759566.934390:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759566.934395:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759566.934396:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759566.934401:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12623 00000100:00080000:1.0:1787759566.934405:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759566.934409:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759566.934419:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759566.934433:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655350144/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759566.934487:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759566/real 1787759566] req@ffff8c05b2d98000 x1874588655350016/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759566.934506:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655350016/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759566.934516:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759566.934519:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759566.934522:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655350016:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787759566.934545:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655350144:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759566.934567:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759566.934576:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759566/real 1787759566] req@ffff8c05b2d99880 x1874588655350144/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759566.934584:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655350144/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759566.934589:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759566.934591:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759566.934593:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655350144:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787759566.934664:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759566.934668:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759566.934669:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759566.934672:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12623 00000100:00080000:1.0:1787759566.934675:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759566.934679:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759566.934682:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759566.934692:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99880 x1874588655350272/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759566.934722:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655350272:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759566.934737:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787759566.934754:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759566.934757:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759566.934758:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759566.934761:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12623 00000100:00080000:1.0:1787759566.934764:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759566.934767:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759566.934770:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759566.934778:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98000 x1874588655350400/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759566.934804:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759566/real 1787759566] req@ffff8c05b2d99880 x1874588655350272/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759566.934812:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99880 x1874588655350272/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759566.934817:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759566.934819:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759566.934821:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655350272:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759566.934829:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655350400:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759566.934848:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759566.934858:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759566/real 1787759566] req@ffff8c05b2d98000 x1874588655350400/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759566.934870:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98000 x1874588655350400/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759566.934901:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759566.934905:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759566.934908:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655350400:192.168.201.107@tcp:8:kworker.0 00000100:00080000:1.0:1787759566.935046:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759566.935051:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759566.935053:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759566.935058:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12623 00000100:00080000:1.0:1787759566.935062:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759566.935067:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759566.935072:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759566.935084:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655350528/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759566.935137:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655350528:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759566.935160:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787759566.935180:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759566.935184:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759566.935186:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759566.935190:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12623 00000100:00080000:1.0:1787759566.935195:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759566.935198:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759566.935207:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759566.935218:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655350656/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759566.935254:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759566/real 1787759566] req@ffff8c05b2d98700 x1874588655350528/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759566.935267:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655350528/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759566.935274:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759566.935277:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759566.935280:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655350528:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759566.935292:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655350656:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759566.935310:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759566.935320:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759566/real 1787759566] req@ffff8c05b2d99f80 x1874588655350656/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759566.935329:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655350656/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759566.935336:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759566.935338:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759566.935341:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655350656:192.168.201.107@tcp:38:kworker.0 00000100:00080000:1.0:1787759566.935427:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759566.935431:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787759566.935432:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759566.935437:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12623 00000100:00080000:1.0:1787759566.935441:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759566.935445:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759566.935450:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759566.935463:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655350784/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759566.935510:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655350784:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759566.935533:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:1.0:1787759566.935554:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759566.935557:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759566.935559:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759566.935563:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12623 00000100:00080000:1.0:1787759566.935567:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759566.935570:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759566.935575:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759566.935587:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655350912/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759566.935618:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759566/real 1787759566] req@ffff8c05b2d9bb80 x1874588655350784/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759566.935635:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655350784/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759566.935642:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759566.935645:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759566.935648:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655350784:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759566.935660:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655350912:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759566.935678:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759566.935689:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759566/real 1787759566] req@ffff8c05b2d9b100 x1874588655350912/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759566.935699:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655350912/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759566.935706:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759566.935709:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759566.935711:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655350912:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759567.062072:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759567.510761:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759568.087274:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759568.534217:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759569.109953:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759569.558129:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759570.134043:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759570.584266:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759571.158047:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759571.606144:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787759572.055390:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759572.055402:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787759572.055407:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759572.055417:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12628 00000100:00080000:3.0:1787759572.055437:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759572.055444:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787759572.055484:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655351040/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759572.055506:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759572.055509:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759572.055511:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759572.055515:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12628 00000100:00080000:3.0:1787759572.055518:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759572.055522:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759572.055530:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759572.055539:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655351168/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759572.055549:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759572.055552:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759572.055554:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759572.055557:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12628 00000100:00080000:3.0:1787759572.055560:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759572.055563:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759572.055567:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759572.055574:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655351296/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759572.055583:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759572.055586:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759572.055588:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759572.055591:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12628 00000100:00080000:3.0:1787759572.055593:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759572.055597:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759572.055601:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759572.055608:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655351424/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759572.055616:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759572.055618:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759572.055620:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759572.055623:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12628 00000100:00080000:3.0:1787759572.055625:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759572.055629:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759572.055632:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759572.055642:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655351552/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759572.055650:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759572.055653:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759572.055654:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759572.055657:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12628 00000100:00080000:3.0:1787759572.055660:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759572.055662:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759572.055666:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759572.055673:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655351680/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759572.055680:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759572.055682:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787759572.055684:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759572.055687:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12628 00000100:00080000:3.0:1787759572.055689:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759572.055692:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759572.055696:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759572.055704:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655351808/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759572.055711:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759572.055714:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759572.055715:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759572.055719:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12628 00000100:00080000:3.0:1787759572.055721:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759572.055724:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759572.055728:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759572.055735:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655351936/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759572.055804:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655351040:192.168.201.107@tcp:250:kworker.0 00000800:00000100:0.0:1787759572.055870:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759572.055951:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759572/real 1787759572] req@ffff8c05a9f7a300 x1874588655351040/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759572.055970:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655351040/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759572.055981:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759572.055985:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759572.055989:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655351040:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787759572.055996:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655351168:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759572.056019:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759572.056049:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759572/real 1787759572] req@ffff8c05a9f79880 x1874588655351168/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759572.056062:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655351168/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759572.056070:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759572.056073:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759572.056076:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655351168:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759572.056081:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655351296:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759572.056100:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759572.056109:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759572/real 1787759572] req@ffff8c05a9f78700 x1874588655351296/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759572.056120:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655351296/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759572.056126:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759572.056130:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759572.056133:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655351296:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759572.056139:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655351424:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759572.056154:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759572.056163:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759572/real 1787759572] req@ffff8c05a9f7b480 x1874588655351424/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759572.056173:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655351424/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759572.056180:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759572.056183:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759572.056186:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655351424:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759572.056191:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655351552:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759572.056207:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759572.056217:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759572/real 1787759572] req@ffff8c05a9f7a680 x1874588655351552/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759572.056226:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655351552/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759572.056234:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759572.056237:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759572.056240:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655351552:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759572.056245:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655351680:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759572.056260:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759572.056269:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759572/real 1787759572] req@ffff8c05a9f78380 x1874588655351680/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759572.056280:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655351680/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759572.056287:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759572.056290:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759572.056293:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655351680:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759572.056298:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655351808:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759572.056315:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759572.056324:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759572/real 1787759572] req@ffff8c05a9f78a80 x1874588655351808/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759572.056335:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655351808/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759572.056342:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759572.056345:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759572.056348:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655351808:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759572.056377:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655351936:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759572.056392:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759572.056401:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759572/real 1787759572] req@ffff8c05a9f7aa00 x1874588655351936/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759572.056411:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655351936/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759572.056424:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759572.056427:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759572.056430:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655351936:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759572.182045:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759572.630343:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759573.206741:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759573.656402:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759574.230226:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759574.678939:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759575.254021:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759575.702133:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759576.278183:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759576.726189:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787759577.174095:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759577.174108:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787759577.174113:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759577.174123:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12633 00000100:00080000:3.0:1787759577.174134:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759577.174141:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787759577.174182:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7aa00 x1874588655352064/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759577.174213:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759577.174217:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759577.174219:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759577.174223:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12633 00000100:00080000:3.0:1787759577.174227:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759577.174231:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759577.174240:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759577.174253:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78a80 x1874588655352192/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759577.174263:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759577.174267:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759577.174269:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759577.174273:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12633 00000100:00100000:0.0:1787759577.174277:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655352064:192.168.201.107@tcp:250:kworker.0 00000100:00080000:3.0:1787759577.174278:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759577.174284:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759577.174290:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759577.174301:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78380 x1874588655352320/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759577.174311:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759577.174314:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759577.174316:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759577.174319:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12633 00000100:00080000:3.0:1787759577.174322:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759577.174326:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759577.174330:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759577.174340:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a680 x1874588655352448/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759577.174347:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759577.174351:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759577.174353:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759577.174356:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12633 00000100:00080000:3.0:1787759577.174359:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759577.174363:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759577.174367:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759577.174377:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b480 x1874588655352576/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759577.174385:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759577.174388:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759577.174390:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759577.174393:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12633 00000800:00000100:0.0:1787759577.174436:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759577.174485:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759577/real 1787759577] req@ffff8c05a9f7aa00 x1874588655352064/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759577.174505:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7aa00 x1874588655352064/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759577.174515:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759577.174519:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759577.174523:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655352064:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787759577.174531:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655352192:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759577.174561:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759577.174575:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759577/real 1787759577] req@ffff8c05a9f78a80 x1874588655352192/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759577.174589:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78a80 x1874588655352192/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759577.174597:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759577.174601:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759577.174605:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655352192:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759577.174631:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655352320:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759577.174650:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759577.174662:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759577/real 1787759577] req@ffff8c05a9f78380 x1874588655352320/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759577.174674:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78380 x1874588655352320/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759577.174682:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759577.174685:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759577.174688:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655352320:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759577.174694:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655352448:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759577.174714:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759577.174725:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759577/real 1787759577] req@ffff8c05a9f7a680 x1874588655352448/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759577.174736:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a680 x1874588655352448/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759577.174743:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759577.174746:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759577.174749:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655352448:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759577.174754:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655352576:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759577.174772:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759577.174783:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759577/real 1787759577] req@ffff8c05a9f7b480 x1874588655352576/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759577.174795:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b480 x1874588655352576/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759577.174803:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759577.174805:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759577.174811:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655352576:192.168.201.107@tcp:38:kworker.0 00000100:00080000:3.0:1787759577.175043:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759577.175048:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759577.175055:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759577.175069:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78700 x1874588655352704/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759577.175093:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759577.175097:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787759577.175099:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759577.175103:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12633 00000100:00080000:3.0:1787759577.175106:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759577.175110:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759577.175115:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759577.175127:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79880 x1874588655352832/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759577.175137:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759577.175141:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759577.175142:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759577.175146:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12633 00000100:00080000:3.0:1787759577.175149:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759577.175153:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759577.175157:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759577.175167:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7a300 x1874588655352960/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759577.175237:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655352704:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759577.175263:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759577.175277:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759577/real 1787759577] req@ffff8c05a9f78700 x1874588655352704/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759577.175290:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78700 x1874588655352704/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759577.175298:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759577.175302:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759577.175305:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655352704:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759577.175312:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655352832:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759577.175331:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759577.175342:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759577/real 1787759577] req@ffff8c05a9f79880 x1874588655352832/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759577.175361:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79880 x1874588655352832/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759577.175369:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759577.175372:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759577.175375:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655352832:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759577.175381:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655352960:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759577.176065:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759577.176080:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759577/real 1787759577] req@ffff8c05a9f7a300 x1874588655352960/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759577.176094:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7a300 x1874588655352960/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759577.176102:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759577.176106:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759577.176109:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655352960:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759577.302282:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759577.750134:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759578.325995:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759578.774145:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759579.350012:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759579.798309:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759580.374004:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759580.822231:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759581.398004:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759581.846133:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787759582.294076:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759582.294087:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787759582.294092:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759582.294103:0:98286:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12638 00000100:00080000:1.0:1787759582.294116:0:98286:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759582.294125:0:98286:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:1.0:1787759582.294168:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b480 x1874588655353088/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759582.294200:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759582.294204:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759582.294207:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759582.294212:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12638 00000100:00080000:1.0:1787759582.294215:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759582.294219:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759582.294229:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759582.294242:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9aa00 x1874588655353216/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759582.294252:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759582.294255:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759582.294257:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759582.294261:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12638 00000100:00080000:1.0:1787759582.294264:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759582.294268:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759582.294273:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759582.294283:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98380 x1874588655353344/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759582.294293:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759582.294296:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759582.294298:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759582.294302:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12638 00000100:00080000:1.0:1787759582.294305:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759582.294309:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759582.294314:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759582.294326:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9a300 x1874588655353472/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759582.294335:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759582.294338:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787759582.294339:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759582.294343:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12638 00000100:00080000:1.0:1787759582.294346:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759582.294350:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759582.294355:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759582.294367:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9b100 x1874588655353600/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759582.294376:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759582.294380:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787759582.294381:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759582.294385:0:98286:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12638 00000100:00080000:1.0:1787759582.294388:0:98286:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759582.294392:0:98286:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759582.294396:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787759582.294407:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d9bb80 x1874588655353728/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759582.294415:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759582.294419:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787759582.294420:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759582.294425:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12638 00000100:00080000:1.0:1787759582.294428:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759582.294432:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759582.294437:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759582.294447:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d99f80 x1874588655353856/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787759582.294456:0:98286:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787759582.294459:0:98286:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787759582.294461:0:98286:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787759582.294464:0:98286:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12638 00000100:00080000:1.0:1787759582.294467:0:98286:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787759582.294470:0:98286:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:1.0:1787759582.294475:0:98286:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787759582.294485:0:98286:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05b2d98700 x1874588655353984/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787759582.294625:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655353088:192.168.201.107@tcp:250:kworker.0 00000800:00000100:0.0:1787759582.294709:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759582.294763:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759582/real 1787759582] req@ffff8c05b2d9b480 x1874588655353088/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759582.294782:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b480 x1874588655353088/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759582.294793:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759582.294797:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759582.294802:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655353088:192.168.201.107@tcp:250:kworker.0 00000100:00100000:0.0:1787759582.294809:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655353216:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759582.294848:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759582.294862:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759582/real 1787759582] req@ffff8c05b2d9aa00 x1874588655353216/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759582.294896:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9aa00 x1874588655353216/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759582.294905:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759582.294908:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759582.294911:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655353216:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759582.294918:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655353344:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759582.294939:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759582.294949:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759582/real 1787759582] req@ffff8c05b2d98380 x1874588655353344/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759582.294962:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98380 x1874588655353344/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759582.294970:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759582.294973:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759582.294977:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655353344:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759582.294983:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655353472:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759582.295001:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759582.295023:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759582/real 1787759582] req@ffff8c05b2d9a300 x1874588655353472/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759582.295035:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9a300 x1874588655353472/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759582.295043:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759582.295046:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759582.295050:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655353472:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759582.295056:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655353600:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759582.295074:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759582.295084:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759582/real 1787759582] req@ffff8c05b2d9b100 x1874588655353600/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759582.295094:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9b100 x1874588655353600/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759582.295103:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759582.295106:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759582.295109:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655353600:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759582.295114:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655353728:192.168.201.107@tcp:38:kworker.0 00000800:00000100:0.0:1787759582.295132:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759582.295141:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759582/real 1787759582] req@ffff8c05b2d9bb80 x1874588655353728/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759582.295153:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d9bb80 x1874588655353728/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759582.295161:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759582.295164:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759582.295167:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d9bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655353728:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759582.295172:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655353856:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759582.295189:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759582.295199:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759582/real 1787759582] req@ffff8c05b2d99f80 x1874588655353856/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759582.295210:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d99f80 x1874588655353856/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759582.295217:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759582.295220:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759582.295223:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d99f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655353856:192.168.201.107@tcp:8:kworker.0 00000100:00100000:0.0:1787759582.295228:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655353984:192.168.201.107@tcp:8:kworker.0 00000800:00000100:0.0:1787759582.295246:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:0.0:1787759582.295256:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759582/real 1787759582] req@ffff8c05b2d98700 x1874588655353984/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759582.295365:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05b2d98700 x1874588655353984/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759582.295378:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787759582.295382:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:0.0:1787759582.295385:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05b2d98700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655353984:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759582.422003:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759582.872260:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759583.446011:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759583.896396:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759584.470979:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759584.918152:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759585.493973:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759585.942721:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759586.517974:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759586.967779:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787759587.415550:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759587.415561:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787759587.415567:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759587.415578:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12643 00000100:00080000:0.0:1787759587.415591:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759587.415598:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:0.0:1787759587.415642:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75880 x1874588655354112/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759587.415675:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759587.415679:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787759587.415681:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759587.415686:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12643 00000100:00080000:0.0:1787759587.415690:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759587.415694:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759587.415705:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759587.415720:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f75c00 x1874588655354240/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759587.415728:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655354112:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787759587.415747:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759587.415760:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787759587.415762:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759587.415767:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12643 00000100:00080000:0.0:1787759587.415771:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759587.415775:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759587.415782:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787759587.415794:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76300 x1874588655354368/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759587.415804:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759587.415808:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787759587.415810:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759587.415814:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12643 00000100:00080000:0.0:1787759587.415818:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759587.415822:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000800:00000100:2.0:1787759587.415829:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000008:00100000:0.0:1787759587.415831:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759587.415845:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76a00 x1874588655354496/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759587.415855:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759587.415859:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787759587.415861:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759587.415865:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12643 00000100:00080000:0.0:1787759587.415869:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787759587.415904:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759587/real 1787759587] req@ffff8c05a9f75880 x1874588655354112/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787759587.415916:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759587.415924:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759587.415926:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75880 x1874588655354112/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787759587.415939:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77800 x1874588655354624/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787759587.415942:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759587.415953:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00080000:0.0:1787759587.415956:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787759587.415961:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655354112:192.168.201.107@tcp:250:kworker.0 00000100:00080000:0.0:1787759587.415963:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787759587.415969:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759587.415975:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12643 00000100:00080000:0.0:1787759587.415979:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759587.415982:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759587.415987:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787759587.415991:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655354240:192.168.201.107@tcp:38:kworker.0 00000100:00100000:0.0:1787759587.416000:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f77b80 x1874588655354752/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759587.416012:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759587.416016:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787759587.416018:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787759587.416024:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00080000:0.0:1787759587.416027:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12643 00000100:00080000:0.0:1787759587.416033:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759587.416037:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759587.416042:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759587.416053:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f76680 x1874588655354880/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787759587.416066:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787759587.416068:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787759587.416070:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787759587.416072:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12643 00000100:00080000:0.0:1787759587.416075:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787759587.416078:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:0.0:1787759587.416082:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787759587.416089:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f74380 x1874588655355008/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759587.416502:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759587/real 1787759587] req@ffff8c05a9f75c00 x1874588655354240/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759587.416520:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f75c00 x1874588655354240/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759587.416530:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759587.416533:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759587.416537:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f75c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655354240:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759587.416578:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655354368:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759587.416619:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759587.416632:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759587/real 1787759587] req@ffff8c05a9f76300 x1874588655354368/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759587.416644:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76300 x1874588655354368/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759587.416651:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759587.416655:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759587.416658:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655354368:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759587.416665:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655354496:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759587.416686:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759587.416697:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759587/real 1787759587] req@ffff8c05a9f76a00 x1874588655354496/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759587.416709:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76a00 x1874588655354496/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759587.416718:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759587.416721:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759587.416725:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655354496:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759587.416731:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655354624:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759587.416763:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759587.416776:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759587/real 1787759587] req@ffff8c05a9f77800 x1874588655354624/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759587.416787:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77800 x1874588655354624/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759587.416796:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759587.416799:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759587.416802:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655354624:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759587.416820:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655354752:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759587.416840:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759587.416851:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759587/real 1787759587] req@ffff8c05a9f77b80 x1874588655354752/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759587.416863:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f77b80 x1874588655354752/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759587.416871:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759587.416873:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759587.416895:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f77b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655354752:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759587.416901:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655354880:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759587.416921:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759587.416933:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759587/real 1787759587] req@ffff8c05a9f76680 x1874588655354880/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759587.416944:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f76680 x1874588655354880/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759587.416951:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759587.416954:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759587.416957:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f76680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655354880:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759587.416963:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655355008:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759587.416979:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759587.416989:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759587/real 1787759587] req@ffff8c05a9f74380 x1874588655355008/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759587.417001:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f74380 x1874588655355008/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759587.417008:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759587.417011:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759587.417014:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f74380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655355008:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759587.542141:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759587.991086:0:15684:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759588.565991:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759589.014127:0:34914:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759589.590535:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759590.040064:0:6171:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759590.614893:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759591.062261:0:49234:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759591.638020:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:3.0:1787759592.086068:0:15681:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787759592.534061:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b3521168-90b1-490d-ad81-e130de8ec2f9->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759592.534072:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787759592.534077:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759592.534089:0:68057:0:(import.c:510:import_select_connection()) MGC192.168.201.107@tcp: connect to NID 192.168.201.107@tcp last attempt 12648 00000100:00080000:3.0:1787759592.534102:0:68057:0:(import.c:589:import_select_connection()) MGC192.168.201.107@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759592.534110:0:68057:0:(import.c:624:import_select_connection()) MGC192.168.201.107@tcp: import ffff8c058596f000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000100:00100000:3.0:1787759592.534157:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79500 x1874588655355136/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759592.534185:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759592.534188:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759592.534190:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759592.534194:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12648 00000100:00080000:3.0:1787759592.534197:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759592.534200:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0589b26800: import ffff8c05891c6800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759592.534209:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759592.534218:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b800 x1874588655355264/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759592.534224:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759592.534227:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759592.534229:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759592.534231:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12648 00000100:00080000:3.0:1787759592.534233:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759592.534235:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0589b26800: import ffff8c05891c0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759592.534237:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759592.534242:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f78e00 x1874588655355392/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759592.534248:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7bc8a071-3537-4c53-83e7-ca9769577496->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759592.534250:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759592.534251:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759592.534253:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: connect to NID 192.168.201.107@tcp last attempt 12648 00000100:00080000:3.0:1787759592.534255:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759592.534257:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0589b26800: import ffff8c0591012800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759592.534260:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759592.534265:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7ad80 x1874588655355520/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759592.534270:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759592.534272:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787759592.534272:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759592.534275:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12648 00000100:00080000:3.0:1787759592.534276:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759592.534279:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8c0583595800: import ffff8c05917d2800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759592.534281:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759592.534289:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7b100 x1874588655355648/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759592.534294:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759592.534296:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787759592.534297:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759592.534299:0:68057:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12648 00000100:00080000:3.0:1787759592.534300:0:68057:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759592.534302:0:68057:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8c0583595800: import ffff8c05917d1000 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759592.534304:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683217 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787759592.534309:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79f80 x1874588655355776/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759592.534313:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759592.534315:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787759592.534316:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759592.534318:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12648 00000100:00080000:3.0:1787759592.534319:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759592.534321:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8c0583595800: import ffff8c05917d4800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759592.534324:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759592.534329:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f7bb80 x1874588655355904/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787759592.534334:0:68057:0:(pinger.c:229:ptlrpc_pinger_process_import()) b211e791-52bd-4fb0-8db7-bbed27b67abc->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787759592.534336:0:68057:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787759592.534336:0:68057:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787759592.534339:0:68057:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: connect to NID 192.168.201.107@tcp last attempt 12648 00000100:00080000:3.0:1787759592.534340:0:68057:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787759592.534342:0:68057:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8c0583595800: import ffff8c05917d0800 using connection 192.168.201.107@tcp/192.168.201.107@tcp 00000008:00100000:3.0:1787759592.534344:0:68057:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683217 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787759592.534348:0:68057:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8c05a9f79180 x1874588655356032/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787759592.534430:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655355136:192.168.201.107@tcp:250:kworker.0 00000800:00000100:2.0:1787759592.534503:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759592.534543:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759592/real 1787759592] req@ffff8c05a9f79500 x1874588655355136/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759592.534555:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79500 x1874588655355136/t0(0) o250->MGC192.168.201.107@tcp@192.168.201.107@tcp:26/25 lens 520/544 e 0 to 1 dl 1787759617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759592.534562:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c058596f000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759592.534565:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759592.534568:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b3521168-90b1-490d-ad81-e130de8ec2f9:2398:1874588655355136:192.168.201.107@tcp:250:kworker.0 00000100:00100000:2.0:1787759592.534573:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655355264:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759592.534587:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759592.534594:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759592/real 1787759592] req@ffff8c05a9f7b800 x1874588655355264/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759592.534601:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b800 x1874588655355264/t0(0) o38->lustre-MDT0000-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759592.534605:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c6800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759592.534607:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759592.534609:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655355264:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759592.534612:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655355392:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759592.534621:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759592.534626:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759592/real 1787759592] req@ffff8c05a9f78e00 x1874588655355392/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759592.534633:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f78e00 x1874588655355392/t0(0) o38->lustre-MDT0001-mdc-ffff8c0589b26800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759592.534644:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05891c0800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759592.534647:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759592.534676:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f78e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655355392:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759592.534682:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655355520:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759592.534702:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759592.534712:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759592/real 1787759592] req@ffff8c05a9f7ad80 x1874588655355520/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759592.534720:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7ad80 x1874588655355520/t0(0) o8->lustre-OST0001-osc-ffff8c0589b26800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759592.534725:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c0591012800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759592.534727:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759592.534729:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7bc8a071-3537-4c53-83e7-ca9769577496:2398:1874588655355520:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759592.534732:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655355648:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759592.534744:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759592.534750:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759592/real 1787759592] req@ffff8c05a9f7b100 x1874588655355648/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759592.534772:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7b100 x1874588655355648/t0(0) o38->lustre-MDT0000-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759592.534780:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759592.534783:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759592.534786:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655355648:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759592.534802:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655355776:192.168.201.107@tcp:38:kworker.0 00000800:00000100:2.0:1787759592.534822:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759592.534833:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759592/real 1787759592] req@ffff8c05a9f79f80 x1874588655355776/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759592.534845:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79f80 x1874588655355776/t0(0) o38->lustre-MDT0001-mdc-ffff8c0583595800@192.168.201.107@tcp:12/10 lens 520/544 e 0 to 1 dl 1787759617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759592.534860:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d1000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759592.534863:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759592.534866:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655355776:192.168.201.107@tcp:38:kworker.0 00000100:00100000:2.0:1787759592.534871:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655355904:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759592.534916:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759592.534924:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759592/real 1787759592] req@ffff8c05a9f7bb80 x1874588655355904/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759592.534931:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f7bb80 x1874588655355904/t0(0) o8->lustre-OST0000-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759592.534936:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d4800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759592.534938:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759592.534939:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f7bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655355904:192.168.201.107@tcp:8:kworker.0 00000100:00100000:2.0:1787759592.534943:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655356032:192.168.201.107@tcp:8:kworker.0 00000800:00000100:2.0:1787759592.534956:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.107@tcp 00000100:00100000:2.0:1787759592.534965:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787759592/real 1787759592] req@ffff8c05a9f79180 x1874588655356032/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787759592.534975:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8c05a9f79180 x1874588655356032/t0(0) o8->lustre-OST0001-osc-ffff8c0583595800@192.168.201.107@tcp:28/4 lens 520/544 e 0 to 1 dl 1787759617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787759592.534984:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8c05917d0800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787759592.534987:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.107@tcp failed (-110) 00000100:00100000:2.0:1787759592.534991:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8c05a9f79180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b211e791-52bd-4fb0-8db7-bbed27b67abc:2398:1874588655356032:192.168.201.107@tcp:8:kworker.0 00000100:00100000:3.0:1787759592.661968:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:0.0:1787759593.110147:0:15683:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759593.686011:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds 00010000:00010000:1.0:1787759594.134284:0:34913:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787759594.710574:0:233583:0:(client.c:2715:ptlrpc_set_wait()) set ffff8c058378a600 going to sleep for 0 seconds Debug log: 76520 lines, 76520 kept, 0 dropped, 0 bad.